Cloud-announce March 2026

cloud-announce@lists.wikimedia.org

3 participants
8 discussions

Proposed updates and changes to cloud-vps 'trove' database as a service
by Andrew Bogott 20 Apr '26

20 Apr '26

tl;dr: new trove database versions are coming! Existing trove db servers will continue to run as before. Please provide feedback about the specifics. -- Sometime in the next few weeks I'd like to make a few updates to the databases supported on cloud-vps. Unless you have your own private database, toolforge users can ignore this email. If you ever visit the 'databases' tab in Horizon please read on! Right now we support the following database engines: mariadb10.5.10 (21 instances) mysql5.7.29(4 instances) postgresql 12.7 (13 instances) I propose to deprecate all three of those engines, and replace them with: mariadb12.2.2 postgres18 Here's how that would look, from a user standpoint: 1) **Any existing database instances will continue to run and operatein Horizonas before.** 2) It will no longer be possible to create new database instances with the deprecated engines. 3) It will no longer be possible to restore backups of databases using the old engine without admin intervention -- if we do this, I will be on hand to do manual restores if necessary. This restriction is bound up with #2 above; either both rules stay or both rules go.[0] 4) There will be no automatic or in-place upgrade path provided from old version to new version. Any upgrades, if needed, will have to be performed 'by hand' with something like an outside-of-openstack dump and restore. This pains me but I've spent some time testing and the internal storage formats are too different for a smooth transition. 5) It will no longer be possible to create new mysql databases. Mysql is not very popular, and as far as I know only provides a functional subset of mariadb. I'm making a lot of assumptions about you, the users, in this proposal, so please speak up if (for example) you perform frequent backup restores, or have a use case that requires mysql. You can respond to this email, or follow up on the associated phab task[1]. If this email provokes only silence or agreement, then I will make these changes on or near April 20. [0] Currently only a total of 9 trove backups exist (and only 4 in the last year), so I suspect no one is really relying on this service on the regular. [1] https://phabricator.wikimedia.org/T420737

1 1

[toolforge] Buildpack environment version upgrade
by David Caro 31 Mar '26

31 Mar '26

Hi! We are going to start rolling out a new version of the buildpack environment for toolforge built images (see the task [1], and the process outline [2]). If you build your container images using the toolforge build service, please keep reading, there might be some actions needed on your part. What you as tool admin need to do: * [optional] For the next two weeks, test your builds using the `toolforge build start --use-latest-versions ...` flag. This will run with the next default version. If you have no errors, no new changes are needed. If you get errors, then you'll need to adapt to the newer versions or open a bug for us to fix it [3]. * [required if you got errors] In two weeks, 31-03-2026, if you want to still use the old deprecated versions of the buildpacks to continue debugging/fixing the builds with the newer versions, use the `toolforge build start --use-deprecated-versions ...` flag. If you were using Aptfile to install packages, you'll have to move to project.toml, see [4]. * [required if you were using the latest versions] That same day, the `--use-latest-versions` flag, will now point to a newer version of the buildpacks, this might break your builds, so you might want to either fix them, open a bug for follow up, and/or use the default version instead. * [required if used the deprecated flag] on 2026-04-27, the flag will be disabled, and you will be required to stop using it. Most of the time there will be no changes needed to your code, but if you were relying on features or old languages that are not supported in the newer buildpacks you might have to adjust your configuration or your code. For reference, the current default version has the buildpacks: heroku_go: 0.1.13 heroku_gradle: 4.0.2 heroku_java: 4.0.2 heroku_jvm: 4.0.2 heroku_maven: 4.0.2 heroku_nodejs: 2.6.2 heroku_nodejs-corepack: 2.6.2 heroku_nodejs-engine: 2.6.2 heroku_nodejs-npm-engine: 2.6.2 heroku_nodejs-npm-install: 2.6.2 heroku_nodejs-pnpm-engine: 2.6.2 heroku_nodejs-pnpm-install: 2.6.2 heroku_nodejs-yarn: 2.6.2 heroku_php: 0.1.2 heroku_procfile: 2.0.2 heroku_python: 0.8.0 heroku_ruby: 2.1.2 heroku_sbt: 4.0.2 heroku_scala: 4.0.2 The current latest version has the buildpacks: heroku_deb-packages: 0.1.3 heroku_dotnet: 0.5.0 heroku_go: 0.5.8 heroku_gradle: 6.2.0 heroku_java: 6.2.0 heroku_jvm: 6.2.0 heroku_maven: 6.2.0 heroku_nodejs: 3.6.0 heroku_nodejs-corepack: 3.6.0 heroku_nodejs-engine: 3.6.0 heroku_nodejs-npm-engine: 3.6.0 heroku_nodejs-npm-install: 3.6.0 heroku_nodejs-pnpm-engine: 3.6.0 heroku_nodejs-pnpm-install: 3.6.0 heroku_nodejs-yarn: 3.6.0 heroku_php: 0.2.4 heroku_procfile: 4.2.1 heroku_python: 0.26.1 heroku_ruby: 7.0.1 heroku_sbt: 6.2.0 heroku_scala: 6.2.0 And the new latest version being rolled out will have the buildpacks: heroku_deb-packages: 0.3.0 heroku_dotnet: 1.0.3 heroku_go: 2.2.1 heroku_gradle: 7.0.9 heroku_java: 7.0.9 heroku_jvm: 7.0.9 heroku_maven: 7.0.9 heroku_nodejs: 5.5.3 heroku_php: 1.4.0 heroku_procfile: 4.2.1 heroku_python: 6.0.1 heroku_ruby: 12.3.0 heroku_sbt: 7.0.9 heroku_scala: 7.0.9 Some improvements are: * Native dotnet support (no need to install external buildpack). * Newer language versions, like: ** dotnet 9.0.203 by default/10.0.103 latest versions ** golang 1.24 default/1.26 latest versions ** jdk versions 1.8.0_432, 11.0.25, 17.0.13, 21.0.5, 23.0.1 by default /1.8.0_482, 11.0.30, 17.0.18, 21.0.10, 25.0.2 latest versions ** php 8.3.7 by default/8.5.3 latest version ** python 3.13 by default with poetry support/3.14 latest version, with uv+poetry support ** ruby 3.3.7 by default/3.3.9 latest version * Native apt buildpack support, if you were using Aptfile, you'll have to change to project.toml [4] You can see upstream the details of each buildpack following the links in [5]. Cheers! [1] https://phabricator.wikimedia.org/T380127 [2] https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Building_co… [3] https://phabricator.wikimedia.org/project/profile/539/ [4] https://github.com/heroku/buildpacks-deb-packages?tab=readme-ov-file#projec… [5] https://github.com/heroku/buildpacks?tab=readme-ov-file#supported-languages -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."

1 3

Cloud VPS network maintenance on March 31st / 7 UTC
by Filippo Giunchedi 30 Mar '26

30 Mar '26

Hello all, as part of improving the Openstack / RabbitMQ operability we will be doing maintenance to the Cloud VPS network starting March 31st at 7 UTC. The work is being tracked in this Phabricator task: https://phabricator.wikimedia.org/T421054 . Thank you for your patience and understanding! best, Filippo -- *Filippo Giunchedi* Staff Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

Re: [Cloud] Re: Proposed updates and changes to cloud-vps 'trove' database as a service
by Andrew Bogott 27 Mar '26

27 Mar '26

Thank you for your thoughts, Travis! As you say, there's unlikely to be any compatibility issues between Trove and the replica databases; in any case, the replicas are due for an upgrade soon. We don't have any plans to actively shutdown older Trove databses. If a situation does arise in the future where that's needed (e.g. a major security concern) I will feel better knowing that alternatives have been available for as long as possible. Running offsite backups is fine (and mostly upside from our perspective) unless you're using a ton of bandwidth and/or your database is extremely large. You shouldn't feel any rush to upgrade your existing DB, although I'm certainly interested in whether or not it is easy for you to restore/import one of your backups onto the new engine. I hope that's all of your concerns, let me know if I missed anything. -Andrew On 3/27/26 8:46 AM, Travis Briggs via Cloud wrote: > Hi Andrew, > > My main concern is that the enwiki_p replica > at enwiki.analytics.db.svc.eqiad.wmflabs is currently on 10.11.13 and > I'm not aware of plans to upgrade it. Tools like ours (mwcurator/WP1) > connect to both the wiki replica and our Trove, and I'm wondering what > the implications of running on different versions would be. > > Answering my own questions in public: > > 1) My general expectation is that newer versions of database software > are backwards compatible from an API perspective, and that differences > are adding features and optimizations. > 2) Keeping in mind the above, I believe both the official MariaDB > connector Python library, as well as pymysql (which is what we > actually use) support both versions. > 3) Having multiple versions for dev environment is not necessary > simply because our databases in dev are containerized. And even then, > it would only be a concern for the most paranoid of development > postures, which we don't take. > > The other thing I worry about is that 10.5 is EOL, and 10.6 is only > supported until July 2026, like you mentioned. In my experience, EOL > is a phrase that makes SREs jumpy. I know you are planning on > continuing to operate existing versions /for now/ but I wonder if > there is a forced migration somewhere on the horizon (see what I did > there?). I'm thinking of the debian upgrades which were fairly painful > for us (though those I understand as being more critical). > > Finally, as a data point for your team, WP1 performs nightly external > backups of our Trove database external to Horizon. I don't know if I > should be mentioning that because we might get in trouble for resource > usage 😅. We have successfully run restore operations in the form of > yearly exercises. > > Thanks as always, > -Travis > > On Fri, Mar 27, 2026 at 6:20 AM Andrew Bogott via Cloud-announce via > Cloud <cloud(a)lists.wikimedia.org> wrote: > > tl;dr: new trove database versions are coming! Existing trove db > servers will continue to run as before. Please provide feedback > about the specifics. > > -- > > Sometime in the next few weeks I'd like to make a few updates to > the databases supported on cloud-vps. Unless you have your own > private database, toolforge users can ignore this email. If you > ever visit the 'databases' tab in Horizon please read on! > > Right now we support the following database engines: > > mariadb10.5.10 (21 instances) > mysql5.7.29(4 instances) > postgresql 12.7 (13 instances) > > I propose to deprecate all three of those engines, and replace > them with: > > mariadb12.2.2 > postgres18 > > Here's how that would look, from a user standpoint: > > 1) **Any existing database instances will continue to run and > operatein Horizonas before.** > > 2) It will no longer be possible to create new database instances > with the deprecated engines. > > 3) It will no longer be possible to restore backups of databases > using the old engine without admin intervention -- if we do this, > I will be on hand to do manual restores if necessary. This > restriction is bound up with #2 above; either both rules stay or > both rules go.[0] > > 4) There will be no automatic or in-place upgrade path provided > from old version to new version. Any upgrades, if needed, will > have to be performed 'by hand' with something like an > outside-of-openstack dump and restore. This pains me but I've > spent some time testing and the internal storage formats are too > different for a smooth transition. > > 5) It will no longer be possible to create new mysql databases. > Mysql is not very popular, and as far as I know only provides a > functional subset of mariadb. > > I'm making a lot of assumptions about you, the users, in this > proposal, so please speak up if (for example) you perform frequent > backup restores, or have a use case that requires mysql. You can > respond to this email, or follow up on the associated phab task[1]. > > If this email provokes only silence or agreement, then I will make > these changes on or near April 20. > > [0] Currently only a total of 9 trove backups exist (and only 4 in > the last year), so I suspect no one is really relying on this > service on the regular. > [1] https://phabricator.wikimedia.org/T420737 > > > > > _______________________________________________ > Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.… > _______________________________________________ > Cloud mailing list -- cloud(a)lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ > > > _______________________________________________ > Cloud mailing list --cloud(a)lists.wikimedia.org > List information:https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimed…

1 0

openstack upgrades Monday, 2026-03-16
by Andrew Bogott 17 Mar '26

17 Mar '26

I will be upgrading the cloud-vps openstack install on Monday, beginning around 17:00 UTC. This will involve one or two brief network resets. There may also be intermittent interruptions to the Horizon UI, and possible issues with scheduling new VMs or creating other cloud-vps resources. Apart from the network resets, toolforge services and existing, running VMs will not be affected. -Andrew + the WMCS team

1 1

Cloud network switch maintenance on Wed March 18th
by Filippo Giunchedi 12 Mar '26

12 Mar '26

Hello all, We will be continuing the work of controlled cloud network switch maintenance on Wednesday March 18th starting at 8 UTC. Expected impact is minimal and we will be monitoring. Thank you for your patience and understanding: controlled tests are an important step towards a more resilient cloud infrastructure. Progress can be followed at https://phabricator.wikimedia.org/T417393 best, Filippo -- *Filippo Giunchedi* Staff Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

Cloud network switch maintenance on Tues March 10th
by Filippo Giunchedi 05 Mar '26

05 Mar '26

Hello all, as part of the effort to improve Cloud availability we will be carrying out controlled network switch testing next week on Tuesday March 10th starting at 8 UTC. Expected impact is limited and we will be monitoring. More details can be found at https://phabricator.wikimedia.org/T417393 best, Filippo -- *Filippo Giunchedi* Staff Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

Old Toolforge jobs will be recreated tomorrow
by David Caro 03 Mar '26

03 Mar '26

Hi all, Tomorrow at approximately 16:00 UTC we will recreate some of the oldest Toolforge jobs to upgrade them to the common job version format. There's no downtime expected and no action is needed on your part, but you might see your jobs disappear and reappear shortly. I will update this thread when we start and when we finish the migration. For details see the Phabricator task[1]. Thanks! ps. for Toolforge feature updates see the changelog [2] [1] https://phabricator.wikimedia.org/T359649 [2] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."

1 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce March 2026