As always, 70% of this conference is about building fresh, new clouds rather than existing use-cases. That made for a very slow start on the first day, but there were some interesting bits later on. Mark Shuttleworth gave a brief talk where he re-affirmed Ubuntu's commitment to supporting OpenStack and K8s in the long-term, and then scolded attendees for getting distracted by (unspecified) shiny new things rather than focusing on the fundamentals. I'm not really sure what that was about but it was nice to hear someone assert that they still think OpenStack is fundamental to the future of cloud tech.
The following is largely notes for my future self, but Brooke might be interested in reading up about Rook.
Ceph/Rook:
Everyone is using ceph! Everyone also talks a lot about how hard it is to deploy. There's a fair amount of buzz around 'Rook' which is a ceph deployment/management system that we might want to consider. As I understand it, you set up a k8s cluster with host networking on all of your OSD nodes, and then Rook dumps a pod on each node which implements the ceph services. Plenty of people are claiming that it works great, and I think it supports rolling upgrades so that might be something to consider instead of a bare puppet-and-debian-package deployment.
Deployment/package management:
There are lots of ways to deploy! Openstack on k8s, openstack on openstack, openstack in containers pushed out by ansible, etc. etc. Almost all of these assume that 1) you're starting from scratch and 2) you want/have ironic control of bare metal. I spent a while thinking that we should set up a k8s cluster and deploy openstack services there... 'airship' might support that model (and it would line up with using Rook to manage the ceph cluster) but I'm not sure that I'm not just looking for a problem to solve when we don't really have one.
The one thing that might be useful for us is grabbing the kolla project packages and deploying on simple standalone docker instances... that would get us out of our current packaging hell. Assuming we don't ever want to patch the projects, this might be a decent alternative to deploying from source.
Designate:
The (two) designate developers are still alive and working on the project. Development is very slow-paced right now, which is mostly good for us because it means fewer headaches during upgrades :) Mugsie (the PTL) switched jobs but says he still has someone paying him to work on the project part-time, so there's no immediate danger of the project dying off.
The Designate folks think that we should keep using designate-sink until we're running version O. Then we can switch to the proper REST-based neutron integration code for creating/deleting records on VM creation and deletion. We'll want to write our own custom Neutron plugin to replace the default one in order to replace the custom code that's currently running in Sink.
The bad news is that the one feature I really want (the ability to share .wmflabs.org between multiple tenants) is on the back-burner for the moment. If money and staff dropped into our lap it might be nice for us to get some contractor dollars devoted to someone working on that (partly because I feel like we're a free-rider on the project and it seems starved for resources).
Keystone:
The keystone upstream is finally implementing system-wide scope for roles, which means that eventually we'll be able to give the 'observer' users a system-wide scope rather than having to add it to every single project. They're also in the process of standardizing on a true project-admin policy which would let us get rid of some of our hacks that allow project admins to add members to their own projects but not others.
Of course, none of that is really useful until other projects have also adopted these concepts, so we won't see any real gains until T or U.
On Thu, May 2, 2019 at 12:25 PM Andrew Bogott abogott@wikimedia.org wrote:
The following is largely notes for my future self, but Brooke might be interested in reading up about Rook.
Ceph/Rook:
Everyone is using ceph! Everyone also talks a lot about how hard it is to deploy.
Yeah, sounds about right...
There's a fair amount of buzz around 'Rook'
I met the maintainers at KubeCon and chatted with them a bit. The biggest drawback I've seen to using it as a ceph deployment was that it isn't typically accessible outside of Kubernetes without using host networking, which didn't seem an option the maintainers were terribly excited about at the time. After following some drama online around that, and figuring the additional burden of configuring a modern-enough Kubernetes on top of Ceph might slow it all down, I'd written it off at that point. They did hit 1.0 four days ago, which is pretty awesome, and it might have become less finicky despite the need for host networking (most issues around that have been closed). Also on the other hand, since I've been in the weeds digging around in requirements for Ceph, I now know that Ceph has issues around Debian kernels that make containers sound absolutely spectacular for deployment at this point (though we'll still have kernel issues on the client side...and if we cannot use the official docker images). Apparently Rook even deploys Nautilus :) We'd be ultra-modern! I'm game for trying it during the PoC at very least. At this point, the production Kubernetes versions are high enough to use it. :)
On the packaging and deploying of OpenStack, kubernetes does sound very "aligned" with other things. Maybe we check if it is horrible for storage first :) We already know what a headache k8s can be, depending on how certs and other things are managed.
Interesting stuff, thanks! When I finally get out of Cleveland, I'll have to write something up for PyCon.
cloud-admin@lists.wikimedia.org