I think it's useful to decouple the topic of 'VMs' from 'OpenStack'. We could conceivably junk everything non-toolforge running on cloud-vps and declare that we're containers-only, but without openstack we would also need to replace or abandon a bunch of other things:
- DNSaaS - DBaaS - Something to manage auth and multi tenancy - Object storage UI - K8s orchestration for paws and superset - Persistent volumes
I can definitely think of piecemeal solutions to each of those, but it seems like a lot of things to re-implement!
I'm curious to hear more about the advantage of promoting containers off of VMs and onto metal. My understanding is that the performance cost of virtualization is very small (although non-0). What are the other advantages of moving containers up to first-class?
-A
On 2/29/24 10:50 AM, David Caro wrote:
Is this to replace toolforge only? Or CloudVPS as a whole?
If CloudVPS, I think that there's many use-cases covered now that would not be possible (or very different) inside k8s, specially for self-managed infra (what would be an openstack project now).
Also, a lot of users are VM-bound, so moving to containers would not be easy for them (or even possible, ex. beta cluster). Not saying if we should or not continue to support those use-cases, just trying to clarify if your proposal is to continue supporting those or not (ex. drop CloudVPS offering and replace with k8s as a service).
On 02/29 18:12, Arturo Borrero Gonzalez wrote:
Hi there,
Last year, we starting evaluating how we could refresh the way we relate (deploy, maintain, upgrade) our Openstack deployment for Cloud VPS [0].
One of the most compelling options we found was to run Openstack inside Kubernetes, using an upstream project called openstack-helm.
But... What if we stopped doing Openstack at all?
To clarify, the base idea I had is:
- deploy Kubernetes to a bunch of hosts in one of our Wikimedia datacenters
** we know how to do it! ** this would be the base, undercloud, or bedrock, whatever.
- deploy ceph next to k8s (maybe, inside even?)
** ceph would remain the preferred network storage solution
- deploy some kind of k8s multiplexing tech
** example: https://www.vcluster.com/ but there could be others ** using this create a dedicated k8s cluster for each project, for example: toolforge/toolsbeta/etc
- Inside this new VM-less toolforge, we can retain pretty much the same
functionalities as today: ** a container listening on 22/tcp with kubectl & toolforge cli installed can be the login bastion ** NFS server can be run on a container, using ceph ** toolsDB can be run on a container. Can't it? Or maybe replace it with other k8s-native solution
- If we need any of the native openstack components, for example Keystone or
Swift we may run them on an standalone fashion inside k8s.
- We already have some base infrastructure (and knowledge) that would
support this model. We have cloudlbs, cloudgw, we know how to do ceph, etc.
- And finally, and most important: the community. The main question could be:
** Is there any software running on Cloud VPS virtual machines that cannot run on a container in kubernetes?
I wanted to start this email hoping that I would collect a list of use cases, blockers, and strong opinions about why running Openstack is important (or not). I'm pretty sure I'm overlooking some important thing.
I plan to document all this on wikitech, and/or maybe phabricator.
You may ask: and why stop doing openstack? I will answer that in a different email to keep this one as short as possible.
Looking forward to your counter-arguments. Thanks!
regards.
[0] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhancemen... _______________________________________________ Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-admin.lists.wikimedia.org/
Cloud-admin mailing list -- cloud-admin@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-admin.lists.wikimedia.org/