Hi there,
Last year, we starting evaluating how we could refresh the way we relate (deploy, maintain, upgrade) our Openstack deployment for Cloud VPS [0].
One of the most compelling options we found was to run Openstack inside Kubernetes, using an upstream project called openstack-helm.
But... What if we stopped doing Openstack at all?
To clarify, the base idea I had is:
* deploy Kubernetes to a bunch of hosts in one of our Wikimedia datacenters ** we know how to do it! ** this would be the base, undercloud, or bedrock, whatever.
* deploy ceph next to k8s (maybe, inside even?) ** ceph would remain the preferred network storage solution
* deploy some kind of k8s multiplexing tech ** example: https://www.vcluster.com/ but there could be others ** using this create a dedicated k8s cluster for each project, for example: toolforge/toolsbeta/etc
* Inside this new VM-less toolforge, we can retain pretty much the same functionalities as today: ** a container listening on 22/tcp with kubectl & toolforge cli installed can be the login bastion ** NFS server can be run on a container, using ceph ** toolsDB can be run on a container. Can't it? Or maybe replace it with other k8s-native solution
* If we need any of the native openstack components, for example Keystone or Swift we may run them on an standalone fashion inside k8s.
* We already have some base infrastructure (and knowledge) that would support this model. We have cloudlbs, cloudgw, we know how to do ceph, etc.
* And finally, and most important: the community. The main question could be: ** Is there any software running on Cloud VPS virtual machines that cannot run on a container in kubernetes?
I wanted to start this email hoping that I would collect a list of use cases, blockers, and strong opinions about why running Openstack is important (or not). I'm pretty sure I'm overlooking some important thing.
I plan to document all this on wikitech, and/or maybe phabricator.
You may ask: and why stop doing openstack? I will answer that in a different email to keep this one as short as possible.
Looking forward to your counter-arguments. Thanks!
regards.
[0] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhancemen...
Is this to replace toolforge only? Or CloudVPS as a whole?
If CloudVPS, I think that there's many use-cases covered now that would not be possible (or very different) inside k8s, specially for self-managed infra (what would be an openstack project now).
Also, a lot of users are VM-bound, so moving to containers would not be easy for them (or even possible, ex. beta cluster). Not saying if we should or not continue to support those use-cases, just trying to clarify if your proposal is to continue supporting those or not (ex. drop CloudVPS offering and replace with k8s as a service).
On 02/29 18:12, Arturo Borrero Gonzalez wrote:
Hi there,
Last year, we starting evaluating how we could refresh the way we relate (deploy, maintain, upgrade) our Openstack deployment for Cloud VPS [0].
One of the most compelling options we found was to run Openstack inside Kubernetes, using an upstream project called openstack-helm.
But... What if we stopped doing Openstack at all?
To clarify, the base idea I had is:
- deploy Kubernetes to a bunch of hosts in one of our Wikimedia datacenters
** we know how to do it! ** this would be the base, undercloud, or bedrock, whatever.
- deploy ceph next to k8s (maybe, inside even?)
** ceph would remain the preferred network storage solution
- deploy some kind of k8s multiplexing tech
** example: https://www.vcluster.com/ but there could be others ** using this create a dedicated k8s cluster for each project, for example: toolforge/toolsbeta/etc
- Inside this new VM-less toolforge, we can retain pretty much the same
functionalities as today: ** a container listening on 22/tcp with kubectl & toolforge cli installed can be the login bastion ** NFS server can be run on a container, using ceph ** toolsDB can be run on a container. Can't it? Or maybe replace it with other k8s-native solution
- If we need any of the native openstack components, for example Keystone or
Swift we may run them on an standalone fashion inside k8s.
- We already have some base infrastructure (and knowledge) that would
support this model. We have cloudlbs, cloudgw, we know how to do ceph, etc.
- And finally, and most important: the community. The main question could be:
** Is there any software running on Cloud VPS virtual machines that cannot run on a container in kubernetes?
I wanted to start this email hoping that I would collect a list of use cases, blockers, and strong opinions about why running Openstack is important (or not). I'm pretty sure I'm overlooking some important thing.
I plan to document all this on wikitech, and/or maybe phabricator.
You may ask: and why stop doing openstack? I will answer that in a different email to keep this one as short as possible.
Looking forward to your counter-arguments. Thanks!
regards.
[0] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhancemen... _______________________________________________ Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-admin.lists.wikimedia.org/
I think it's useful to decouple the topic of 'VMs' from 'OpenStack'. We could conceivably junk everything non-toolforge running on cloud-vps and declare that we're containers-only, but without openstack we would also need to replace or abandon a bunch of other things:
- DNSaaS - DBaaS - Something to manage auth and multi tenancy - Object storage UI - K8s orchestration for paws and superset - Persistent volumes
I can definitely think of piecemeal solutions to each of those, but it seems like a lot of things to re-implement!
I'm curious to hear more about the advantage of promoting containers off of VMs and onto metal. My understanding is that the performance cost of virtualization is very small (although non-0). What are the other advantages of moving containers up to first-class?
-A
On 2/29/24 10:50 AM, David Caro wrote:
Is this to replace toolforge only? Or CloudVPS as a whole?
If CloudVPS, I think that there's many use-cases covered now that would not be possible (or very different) inside k8s, specially for self-managed infra (what would be an openstack project now).
Also, a lot of users are VM-bound, so moving to containers would not be easy for them (or even possible, ex. beta cluster). Not saying if we should or not continue to support those use-cases, just trying to clarify if your proposal is to continue supporting those or not (ex. drop CloudVPS offering and replace with k8s as a service).
On 02/29 18:12, Arturo Borrero Gonzalez wrote:
Hi there,
Last year, we starting evaluating how we could refresh the way we relate (deploy, maintain, upgrade) our Openstack deployment for Cloud VPS [0].
One of the most compelling options we found was to run Openstack inside Kubernetes, using an upstream project called openstack-helm.
But... What if we stopped doing Openstack at all?
To clarify, the base idea I had is:
- deploy Kubernetes to a bunch of hosts in one of our Wikimedia datacenters
** we know how to do it! ** this would be the base, undercloud, or bedrock, whatever.
- deploy ceph next to k8s (maybe, inside even?)
** ceph would remain the preferred network storage solution
- deploy some kind of k8s multiplexing tech
** example: https://www.vcluster.com/ but there could be others ** using this create a dedicated k8s cluster for each project, for example: toolforge/toolsbeta/etc
- Inside this new VM-less toolforge, we can retain pretty much the same
functionalities as today: ** a container listening on 22/tcp with kubectl & toolforge cli installed can be the login bastion ** NFS server can be run on a container, using ceph ** toolsDB can be run on a container. Can't it? Or maybe replace it with other k8s-native solution
- If we need any of the native openstack components, for example Keystone or
Swift we may run them on an standalone fashion inside k8s.
- We already have some base infrastructure (and knowledge) that would
support this model. We have cloudlbs, cloudgw, we know how to do ceph, etc.
- And finally, and most important: the community. The main question could be:
** Is there any software running on Cloud VPS virtual machines that cannot run on a container in kubernetes?
I wanted to start this email hoping that I would collect a list of use cases, blockers, and strong opinions about why running Openstack is important (or not). I'm pretty sure I'm overlooking some important thing.
I plan to document all this on wikitech, and/or maybe phabricator.
You may ask: and why stop doing openstack? I will answer that in a different email to keep this one as short as possible.
Looking forward to your counter-arguments. Thanks!
regards.
[0] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhancemen... _______________________________________________ Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-admin.lists.wikimedia.org/
Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-admin.lists.wikimedia.org/
On 2/29/24 19:23, Andrew Bogott wrote:
we're containers-only, but without openstack we would also need to replace or abandon a bunch of other things:
- DNSaaS
I have been thinking about this as well. Most of the DNSaaS usage is because nova, isn't it?
I believe the DNS abstraction + integration that kubernetes has via Services resource is very powerful. That, with the ingress, and the external ops/dns.git may cover all our use cases.
Can you give an example of a DNS entry that we would loose if going containers only?
- DBaaS
This is 100% true.
I honestly ignore if:
a) trove is capable of scheduling DBs as containers b) there is a k8s-native DBaaS solution
- Something to manage auth and multi tenancy
The auth/multitenancy in Toolforge is done via LDAP/Striker/maintain-kubeusers, no? Keystone has very little role in Toolforge.
- Object storage UI
This is 100% true.
Also, this is maybe not the most difficult thing to implement ourselves, if we really need this at all.
- Persistent volumes
I think we may implement similar semantics using k8s PV/PVCs, but I could be wrong.
I'm curious to hear more about the advantage of promoting containers off of VMs and onto metal. My understanding is that the performance cost of virtualization is very small (although non-0). What are the other advantages of moving containers up to first-class?
Some ideas:
== easier maintenance and operation ==
My experience is that maintaining and operating a k8s cluster is way easier than maintaining and operating an openstack cluster.
Upgrades are faster. Components are simpler. They break less.
I can't count how many engineering hours we have spent dealing with rabbitmq problems, or galera problems, or some random openstack API problem.
We just don't seem to have any of this with k8s.
Don't be afraid to tell me if I'm being biased here.
== easier for users ==
Today, use-cases that don't fit in Toolforge are told to move into Cloud VPS. This implies a full system administration that for most cases don't need to be. A container enginer with less platform restrictions than Toolforge may suffice in some cases.
I believe that, in the long run, the community may benefit from having a Toolforge fall back to be a managed k8s, rather than a VM hosting.
== storage may be lighter for containers ==
If I'm not mistaken, in ceph we don't do any de-duplication for common blocks. This means that we store things like linux kernel images for each VM instance we have. Well, the whole base system.
If the debian base system is 1G and we have 1000 VMs, that's 1 TB of data. With 3 ceph copies, 3 TB worth of hardware.
Storage for container images may be lighter if we compare to this, due to its layered filesystem nature.
This is just a theory anyway.
== futuristic tech vs old tech ==
Containers feels the future. VMs feels as old tech. I know this is not a very elaborated argument, but is just a feeling I have from time to time.
On 2/29/24 10:50 AM, David Caro wrote:
Is this to replace toolforge only? Or CloudVPS as a whole?
Yes, CloudVPS as a whole. Stop offering VM hosting, and only offer Container hosting.
If CloudVPS, I think that there's many use-cases covered now that would not be possible (or very different) inside k8s, specially for self-managed infra (what would be an openstack project now).
I would like to collect such list for future reference.
Also, a lot of users are VM-bound, so moving to containers would not be easy for them (or even possible, ex. beta cluster). Not saying if we should
Not sure if the beta cluster is a good example in this context.
Could we find any other strongly VM-bound users?
proposal is to continue supporting those or not (ex. drop CloudVPS offering and replace with k8s as a service).
Maybe we could aim to offer 3 levels of k8s:
* via toolforge ** the PaaS we know, the most abstracted, via the APIs we are creating
* via managed k8s ** you get a dedicated namespace for you, unrestricted within the defined limits
* via self managed k8s ** you get a whole cluster (a virtual one, again, maybe via vcluster or similar)
Thank you for starting the discussion! I'll start by saying that while I'm not convinced by the arguments for moving everything from OpenStack to containers, I tend to agree that OpenStack does have some features we should reconsider whether we want to offer them. For example, my current feeling is that the half-baked PostgreSQL support in Trove is a net negative considering incidents like [0] which both take a significant amount of admin time to troubleshoot and resolve and as a result are causing quite a bit of downtime for our users.
[0]: https://phabricator.wikimedia.org/T355138
First, I am highly sceptical of the Kubernetes cluster-in-cluster solutions you mention. As far as we should be concerned as infrastructure operators, having the ability to run arbitrary workloads in a Kubernetes cluster is equivalent to full root on all the worker nodes. (The most obvious example is the ability to run a pod as root that mounts /etc/passwd or a similarly privileged file as read-write as a hostPath, but that's far from the only way to break out of the pod sandbox.) We solve this in Toolforge with PSPs that severely restrict the configurations of pods that are allowed to run. I am having a hard time imagining how to implement a shared cluster without those Toolforge-level restrictions that would keep that strong tenant isolation that at least I consider a hard requirement for any of our offerings.
From my experience maintaining the K8s cluster in Toolforge, I can say that upgrading that cluster is consistently one of the most stressful things I do around here, and that's with the majority of the workload being managed by us. Yes, the process is rather simple now, but compared to OpenStack, Kubernetes has a very active role in the continued running of already existing workloads. The blast radius of a Kubernetes upgrade going wrong is much higher than, say, a Nova upgrade going wrong which would affect starting new VMs and stopping existing ones but generally would not touch VMs already running in libvirt. So far there's only been one (if I remember correctly) upgrade-related major service degradation that made it to live Toolforge[1], but I would credit that more to the slow speed of our upgrades and the countless number of hours I've spent reading changelogs and docs and testing the upgrades locally and in toolsbeta. And as a reminder, we're currently about two years behind Kubernetes releases and don't seem to be catching up even after upstream reduced from 4 1.x releases a year to 3.
[1]: https://phabricator.wikimedia.org/T308189
The fact that Toolforge K8s runs in VMs is very helpful due to the flexibility it gives - if I want to test a particular combination of worker configuration I can currently just spin up a VM instead of having to figure out where to find hardware for that (like I'm currently having to think for the OVS tests). Also, many projects are just too small to need dedicated hardware. For example, LibUp[2], a project I recently became involved with and that doesn't neatly fit into Toolforge at the moment, currently uses about 10 vCPUs and about 20 GiB of RAM - we can stuff about maybe 10-20 of projects of that size onto 1U of rack space on a modern high-spec virtualization node so giving it dedicated hardware to run a K8s cluster on top of just does not make sense. And yes, you could replace OpenStack with something like Ganeti with much less management overhead, but my gut feeling is that Nova and Neutron and the other 'core' services involved in running traditional VMs are relatively well-behaving compared to the newer stuff, and also give us useful features (like multi-tenancy, and instance isolation from the management/wikiland networks) that we'd have to invent ourselves if we ditched OpenStack.
[2]: https://www.mediawiki.org/wiki/LibUp
And finally, I disagree with the statement that maintaining a Linux server is more difficult than running something in Kubernetes (even if someone else maintains the cluster itself). At least in my mind a modern Kubernetes deployment has a million more moving parts than a simple Linux server where our users can SSH to and apt-get install a web server to run their app with.
Taavi
On Thu, Feb 29, 2024 at 7:12 PM Arturo Borrero Gonzalez aborrero@wikimedia.org wrote:
Hi there,
Last year, we starting evaluating how we could refresh the way we relate (deploy, maintain, upgrade) our Openstack deployment for Cloud VPS [0].
One of the most compelling options we found was to run Openstack inside Kubernetes, using an upstream project called openstack-helm.
But... What if we stopped doing Openstack at all?
To clarify, the base idea I had is:
- deploy Kubernetes to a bunch of hosts in one of our Wikimedia datacenters
** we know how to do it! ** this would be the base, undercloud, or bedrock, whatever.
- deploy ceph next to k8s (maybe, inside even?)
** ceph would remain the preferred network storage solution
- deploy some kind of k8s multiplexing tech
** example: https://www.vcluster.com/ but there could be others ** using this create a dedicated k8s cluster for each project, for example: toolforge/toolsbeta/etc
- Inside this new VM-less toolforge, we can retain pretty much the same
functionalities as today: ** a container listening on 22/tcp with kubectl & toolforge cli installed can be the login bastion ** NFS server can be run on a container, using ceph ** toolsDB can be run on a container. Can't it? Or maybe replace it with other k8s-native solution
- If we need any of the native openstack components, for example Keystone or
Swift we may run them on an standalone fashion inside k8s.
- We already have some base infrastructure (and knowledge) that would support
this model. We have cloudlbs, cloudgw, we know how to do ceph, etc.
- And finally, and most important: the community. The main question could be:
** Is there any software running on Cloud VPS virtual machines that cannot run on a container in kubernetes?
I wanted to start this email hoping that I would collect a list of use cases, blockers, and strong opinions about why running Openstack is important (or not). I'm pretty sure I'm overlooking some important thing.
I plan to document all this on wikitech, and/or maybe phabricator.
You may ask: and why stop doing openstack? I will answer that in a different email to keep this one as short as possible.
Looking forward to your counter-arguments. Thanks!
regards.
[0] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhancemen... _______________________________________________ Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-admin.lists.wikimedia.org/
cloud-admin@lists.wikimedia.org