Hi there,
This contains information on the projects that I have been working on lately and how to continue work if necessary for any continuity reasons. I figured I would send an email because I'm going to be on PTO (vacations) for the next 5 weeks. I will be back to work on 2023-09-04.
** cloudlb project @ eqiad1 [1] Current status is that cloudcontrol1005 is already in the new setup. We are waiting for cloudservices1006 to be racked [2] before doing anything else. Cloudservices1006 should go directly into the new .eqiad.wmnet setup with cloud-private and BGP-based VIPs et all. I bootstraped the config (even if the server is not there yet) in a gerrit patch [3], to be merged when the server is in place.
The idea is that cloudcontrol1005+cloudservices1006 will make for a tiny openstack control plane, enough for us to move the 'openstack.eqiad1.wikimediacloud.org' endpoint to cloudlb1001/cloudlb1002 (BGP VIP). Also, to move all the VMs to the new DNS servers [4].
Once this endpoint is migrated, we should decom / rerack / rename / rerack the rest of cloudcontrols and cloudservices1005 according to the plans [5], therefore scaling up the newer cloud-private-based control plane.
This is exactly the codfw1dev setup, and it works really well as a setup verification / comparison.
This is a quarter KR/goal. You (Andrew?) should feel free to take over this one or wait until I'm back. Cathal @ NetOps knows this project in deep, including the network implementation details, and should be available to help and assist as required.
The work here is exciting and will also open the door to work on the kubernetes undercloud [10] (see below).
** Toolforge build service [6]. I was trying to get this patch developed, but got into the rabbit hole of making the buildservice development environment setup reproducible using lima-kilo [7]. This went surprisingly well, and the only missing bit is what to do with harbor [8], which may not even be a blocker to develop the builds API code itself. This also touches on the helm vs secrets problem.
Bonus point is that lima-kilo also helped me migrate the jobs-framework-emailer to the new toolforge CI/CD setup "easily" [9].
Looking forward to continue work in this space, but David/Raymond should feel free to take over as required / desired.
** kubernetes undercloud @ codfw [10]. Not much here at the moment, but we are approaching the point in which hardware will be available in codfw for us to start playing [11]. Cathal, Nicholas and I already shared a few comments on IRC about switches, rack footprint, etc. I belive if we keep pushing in the right direction, we may hit the ground running and get a POC bootstrapped in september ??
To be clear: DON'T use the hardware to refresh codfw1dev. Think first if we can use it to build codfw2dev (or whatever the name). Or maybe refresh codfw1dev but don't decomm the replaced hardware just yet. We'll need a buffer of hardware to play with kubernetes and openstack-helm.
We don't even have an explicit KR/Goal for this in this quarter, but definitely worth keeping in the radar given the hardware is arriving soon.
** Toolforge kubernetes upgrade to 1.23 [12]. This is a quarter KR / goal. But nothing done in this space. Taavi should feel free to continue without me if desired / required.
** Finally, you can check my personal phabricator workboard [0]. I track all my tasks in there.
regards.
[0] https://phabricator.wikimedia.org/tag/user-aborrero/ [1] https://phabricator.wikimedia.org/T341060 [2] https://phabricator.wikimedia.org/T342161 [3] https://gerrit.wikimedia.org/r/c/941383 [4] https://phabricator.wikimedia.org/T342621 [5] https://phabricator.wikimedia.org/T341494 [6] https://phabricator.wikimedia.org/T340031 [7] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/lima-k... [8] https://phabricator.wikimedia.org/T342853 [9] https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-framework-emailer [10] https://phabricator.wikimedia.org/T342750 [11] https://phabricator.wikimedia.org/T342456 [12] https://phabricator.wikimedia.org/T298005