Cloud March 2026

cloud@lists.wikimedia.org

11 participants
10 discussions

[Cloud-announce] Proposed updates and changes to cloud-vps 'trove' database as a service
by Andrew Bogott via Cloud-announce 20 Apr '26

20 Apr '26

tl;dr: new trove database versions are coming! Existing trove db servers will continue to run as before. Please provide feedback about the specifics. -- Sometime in the next few weeks I'd like to make a few updates to the databases supported on cloud-vps. Unless you have your own private database, toolforge users can ignore this email. If you ever visit the 'databases' tab in Horizon please read on! Right now we support the following database engines: mariadb10.5.10 (21 instances) mysql5.7.29(4 instances) postgresql 12.7 (13 instances) I propose to deprecate all three of those engines, and replace them with: mariadb12.2.2 postgres18 Here's how that would look, from a user standpoint: 1) **Any existing database instances will continue to run and operatein Horizonas before.** 2) It will no longer be possible to create new database instances with the deprecated engines. 3) It will no longer be possible to restore backups of databases using the old engine without admin intervention -- if we do this, I will be on hand to do manual restores if necessary. This restriction is bound up with #2 above; either both rules stay or both rules go.[0] 4) There will be no automatic or in-place upgrade path provided from old version to new version. Any upgrades, if needed, will have to be performed 'by hand' with something like an outside-of-openstack dump and restore. This pains me but I've spent some time testing and the internal storage formats are too different for a smooth transition. 5) It will no longer be possible to create new mysql databases. Mysql is not very popular, and as far as I know only provides a functional subset of mariadb. I'm making a lot of assumptions about you, the users, in this proposal, so please speak up if (for example) you perform frequent backup restores, or have a use case that requires mysql. You can respond to this email, or follow up on the associated phab task[1]. If this email provokes only silence or agreement, then I will make these changes on or near April 20. [0] Currently only a total of 9 trove backups exist (and only 4 in the last year), so I suspect no one is really relying on this service on the regular. [1] https://phabricator.wikimedia.org/T420737 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

3 3

[Cloud-announce] [toolforge] Buildpack environment version upgrade
by David Caro via Cloud-announce 31 Mar '26

31 Mar '26

Hi! We are going to start rolling out a new version of the buildpack environment for toolforge built images (see the task [1], and the process outline [2]). If you build your container images using the toolforge build service, please keep reading, there might be some actions needed on your part. What you as tool admin need to do: * [optional] For the next two weeks, test your builds using the `toolforge build start --use-latest-versions ...` flag. This will run with the next default version. If you have no errors, no new changes are needed. If you get errors, then you'll need to adapt to the newer versions or open a bug for us to fix it [3]. * [required if you got errors] In two weeks, 31-03-2026, if you want to still use the old deprecated versions of the buildpacks to continue debugging/fixing the builds with the newer versions, use the `toolforge build start --use-deprecated-versions ...` flag. If you were using Aptfile to install packages, you'll have to move to project.toml, see [4]. * [required if you were using the latest versions] That same day, the `--use-latest-versions` flag, will now point to a newer version of the buildpacks, this might break your builds, so you might want to either fix them, open a bug for follow up, and/or use the default version instead. * [required if used the deprecated flag] on 2026-04-27, the flag will be disabled, and you will be required to stop using it. Most of the time there will be no changes needed to your code, but if you were relying on features or old languages that are not supported in the newer buildpacks you might have to adjust your configuration or your code. For reference, the current default version has the buildpacks: heroku_go: 0.1.13 heroku_gradle: 4.0.2 heroku_java: 4.0.2 heroku_jvm: 4.0.2 heroku_maven: 4.0.2 heroku_nodejs: 2.6.2 heroku_nodejs-corepack: 2.6.2 heroku_nodejs-engine: 2.6.2 heroku_nodejs-npm-engine: 2.6.2 heroku_nodejs-npm-install: 2.6.2 heroku_nodejs-pnpm-engine: 2.6.2 heroku_nodejs-pnpm-install: 2.6.2 heroku_nodejs-yarn: 2.6.2 heroku_php: 0.1.2 heroku_procfile: 2.0.2 heroku_python: 0.8.0 heroku_ruby: 2.1.2 heroku_sbt: 4.0.2 heroku_scala: 4.0.2 The current latest version has the buildpacks: heroku_deb-packages: 0.1.3 heroku_dotnet: 0.5.0 heroku_go: 0.5.8 heroku_gradle: 6.2.0 heroku_java: 6.2.0 heroku_jvm: 6.2.0 heroku_maven: 6.2.0 heroku_nodejs: 3.6.0 heroku_nodejs-corepack: 3.6.0 heroku_nodejs-engine: 3.6.0 heroku_nodejs-npm-engine: 3.6.0 heroku_nodejs-npm-install: 3.6.0 heroku_nodejs-pnpm-engine: 3.6.0 heroku_nodejs-pnpm-install: 3.6.0 heroku_nodejs-yarn: 3.6.0 heroku_php: 0.2.4 heroku_procfile: 4.2.1 heroku_python: 0.26.1 heroku_ruby: 7.0.1 heroku_sbt: 6.2.0 heroku_scala: 6.2.0 And the new latest version being rolled out will have the buildpacks: heroku_deb-packages: 0.3.0 heroku_dotnet: 1.0.3 heroku_go: 2.2.1 heroku_gradle: 7.0.9 heroku_java: 7.0.9 heroku_jvm: 7.0.9 heroku_maven: 7.0.9 heroku_nodejs: 5.5.3 heroku_php: 1.4.0 heroku_procfile: 4.2.1 heroku_python: 6.0.1 heroku_ruby: 12.3.0 heroku_sbt: 7.0.9 heroku_scala: 7.0.9 Some improvements are: * Native dotnet support (no need to install external buildpack). * Newer language versions, like: ** dotnet 9.0.203 by default/10.0.103 latest versions ** golang 1.24 default/1.26 latest versions ** jdk versions 1.8.0_432, 11.0.25, 17.0.13, 21.0.5, 23.0.1 by default /1.8.0_482, 11.0.30, 17.0.18, 21.0.10, 25.0.2 latest versions ** php 8.3.7 by default/8.5.3 latest version ** python 3.13 by default with poetry support/3.14 latest version, with uv+poetry support ** ruby 3.3.7 by default/3.3.9 latest version * Native apt buildpack support, if you were using Aptfile, you'll have to change to project.toml [4] You can see upstream the details of each buildpack following the links in [5]. Cheers! [1] https://phabricator.wikimedia.org/T380127 [2] https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Building_co… [3] https://phabricator.wikimedia.org/project/profile/539/ [4] https://github.com/heroku/buildpacks-deb-packages?tab=readme-ov-file#projec… [5] https://github.com/heroku/buildpacks?tab=readme-ov-file#supported-languages -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 3

[Cloud-announce] Cloud VPS network maintenance on March 31st / 7 UTC
by Filippo Giunchedi via Cloud-announce 30 Mar '26

30 Mar '26

Hello all, as part of improving the Openstack / RabbitMQ operability we will be doing maintenance to the Cloud VPS network starting March 31st at 7 UTC. The work is being tracked in this Phabricator task: https://phabricator.wikimedia.org/T421054 . Thank you for your patience and understanding! best, Filippo -- *Filippo Giunchedi* Staff Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/> _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

Toolforge "sandbox" project for beginners?
by Travis Briggs 27 Mar '26

27 Mar '26

Hello all, The Bay Area Wikipedians User Group organizers (BAWUG: https://meta.wikimedia.org/wiki/Bay_Area_Wikipedians_User_Group), is in the early stages of planning a satellite "hackathon" to coincide with the main one in Milan in May. As part of the planning, I'm thinking about doing more of a "beginner's workshop" type format, where there is a lot of hands on training and we work as a group on one or two projects. This is in contrast to the normal hackathon format where you give suggestions then set a bunch of people loose to work individually and in teams on a variety of projects. To be clear, we're anticipating 10 or less attendees, with almost no experience in Wikimedia development (but still developers of at least moderate other experience) Anyways, I was wondering if there's any capacity for us to have a "sandbox" type toolforge project, where we can get our participants onboarded with developer accounts, and they can experiment with developing and launching some kind of "hello world" tool. I think it would be a great experience to set up the accounts, the keys, the SSH tunnels, etc, as well as learning how to query enwiki_p and access other resources. Though looking through the onboarding docs just now (it's been ages since I've done this process myself and it clearly has changed), it says "Toolforge membership" can take up to a week for approval? This might not be an issue because I was already planning to have a "pre-session" for our workshop a week before the in person event. Basically my request comes down to: Can I just go ahead and use the self service tool account creation workflow ( https://wikitech.wikimedia.org/wiki/Help:Toolforge/Tool_accounts#Creating) with the knowledge that I'm not developing a "real" tool, it's for educational purposes? Thanks, -Travis BAWUG Co-organizer

5 6

Toolforge job stuck at "Run not attempted yet. Pod in 'Pending' phase."
by Sebastian Berlin 23 Mar '26

23 Mar '26

A Toolforge job I'm running is stuck. I started it last Friday after having stopped once before for now apparent reason. When I checked today I got: > tools.commonsdb-registry@tools-bastion-15:~$ toolforge jobs show > make-declarations-wlm-2013 > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Job name: | make-declarations-wlm-2013 > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Command: | src/make_declaration.py -t mem:1.0Gi -t cpu:3.0 -t > version:0.1.11 --verbose batch:category-26152515 | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Job type: | one-off > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Image: | tool-commonsdb-registry/tool-commonsdb-registry:latest > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Port: | none > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | File log: | yes > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Output log: | > /data/project/commonsdb-registry/make-declarations-wlm-2013.out > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Error log: | > /data/project/commonsdb-registry/make-declarations-wlm-2013.err > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Emails: | all > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Resources: | mem: 1.0Gi, cpu: 3.0 > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Replicas: | > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Mounts: | all > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Retry: | no > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Timeout: | no > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Health check: | none > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Status: | Running for 3d23m > | > > +---------------+-----------------------------------------------------------------------------------------------------+ > | Hints: | Run not attempted yet. Pod in 'Pending' phase. > | > > +---------------+-----------------------------------------------------------------------------------------------------+ As far as I can tell that means it hasn't actually started. It's still waiting for something, but it's unclear what. I restarted it a couple of times today so far, but it never gets past that stage. When I've run similar jobs in the past they sometimes take a while to start. Eventually they have though, so I though maybe they were waiting for resources or something. Can anyone help me get this started or explain if I've misunderstood something? *Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE) E-post/*E-Mail*: sebastian.berlin(a)wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84

3 3

[Cloud-announce] openstack upgrades Monday, 2026-03-16
by Andrew Bogott via Cloud-announce 17 Mar '26

17 Mar '26

I will be upgrading the cloud-vps openstack install on Monday, beginning around 17:00 UTC. This will involve one or two brief network resets. There may also be intermittent interruptions to the Horizon UI, and possible issues with scheduling new VMs or creating other cloud-vps resources. Apart from the network resets, toolforge services and existing, running VMs will not be affected. -Andrew + the WMCS team _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

[Cloud-announce] Cloud network switch maintenance on Wed March 18th
by Filippo Giunchedi via Cloud-announce 12 Mar '26

12 Mar '26

Hello all, We will be continuing the work of controlled cloud network switch maintenance on Wednesday March 18th starting at 8 UTC. Expected impact is minimal and we will be monitoring. Thank you for your patience and understanding: controlled tests are an important step towards a more resilient cloud infrastructure. Progress can be followed at https://phabricator.wikimedia.org/T417393 best, Filippo -- *Filippo Giunchedi* Staff Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/> _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

[Cloud-announce] Cloud network switch maintenance on Tues March 10th
by Filippo Giunchedi via Cloud-announce 05 Mar '26

05 Mar '26

Hello all, as part of the effort to improve Cloud availability we will be carrying out controlled network switch testing next week on Tuesday March 10th starting at 8 UTC. Expected impact is limited and we will be monitoring. More details can be found at https://phabricator.wikimedia.org/T417393 best, Filippo -- *Filippo Giunchedi* Staff Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/> _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

[Cloud-announce] Old Toolforge jobs will be recreated tomorrow
by David Caro via Cloud-announce 03 Mar '26

03 Mar '26

Hi all, Tomorrow at approximately 16:00 UTC we will recreate some of the oldest Toolforge jobs to upgrade them to the common job version format. There's no downtime expected and no action is needed on your part, but you might see your jobs disappear and reappear shortly. I will update this thread when we start and when we finish the migration. For details see the Phabricator task[1]. Thanks! ps. for Toolforge feature updates see the changelog [2] [1] https://phabricator.wikimedia.org/T359649 [2] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 2

Fwd: [Wikitech-l] New global API rate limits
by Pine W 02 Mar '26

02 Mar '26

Hi, I'm forwarding the below to additional email lists in case the announcement interests additional people who aren't subscribed to Wiktech-l. I suggest that any questions or comments for this topic which are intended for public email discussion be placed in the original thread on Wikitech-l Regards, Pine🌲 ---------- Forwarded message --------- From: Jonathan Tweed via Wikitech-l <wikitech-l(a)lists.wikimedia.org> Date: Mon, Mar 2, 2026 at 8:44 AM Subject: [Wikitech-l] New global API rate limits To: <wikitech-l(a)lists.wikimedia.org> Cc: Jonathan Tweed <jtweed(a)wikimedia.org> Hi all To help ensure fair and sustainable access <https://www.mediawiki.org/wiki/MediaWiki_Product_Insights/Content_Reuse> to Wikimedia resources, over the next month the Wikimedia Foundation will implement global API rate limits across our APIs. Why we’re doing this As we’ve shared over the last year <https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of…>, since 2024 we’ve observed a significant rise in automated requests, across scraping, APIs and bulk downloads. This is continuing to cause unsustainable load on our infrastructure, taking time and resources away that we need to support the Wikimedia projects, contributors and readers. To ensure we can continue to provide preferential access for human and mission-oriented traffic, we need to reduce the amount of unidentified requests to our APIs from its current level of around 33%. This reduction will also enable us to improve governance around fair use, in line with our API Usage Guidelines <https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_API_Usage…> . What we’re doing In early March, we will apply low limits to anonymous API requests that originate from outside Toolforge/WMCS and requests that are made from web browsers. In early April, higher limits will be applied to identified traffic, including authenticated requests. Authenticated requests from a user in the ‘bot’ user group on any wiki will not be subject to these new limits, nor will clients which are well known to the Wikimedia Foundation. API requests from Toolforge/WMCS will also be exempt from rate limits for now. Regardless, all developers are encouraged to familiarise themselves with the new limits and associated best practices to ensure they are not misclassified as abusive traffic. You can find this information at Wikimedia APIs/Rate limits <https://www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits> . What this means in practice Tools that use Wikimedia-hosted APIs, including Gadgets that make API requests, may be subject to new cross-API limits that are global across all Wikimedia projects. These limits are intentionally set as high as possible to minimise impact on the community. We are asking developers to identify their requests to access higher limits. Tools running in Toolforge/WMCS are exempted for now, but we request that you follow the User-Agent policy <https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agen…> and provide a meaningful User-Agent to help us correctly identify the source of traffic. Otherwise, authenticating using session cookies or OAuth 2.0 will grant a higher limit. Where the authenticated user has the community approved bot right on any wiki, this will also exempt you from limits, even when the tool is not running on Toolforge/WMCS. This request for authentication is a change from the previous guidance in the Robot policy, which suggested not authenticating to improve cache hit rates. We do not expect any significant impact from this change with current usage patterns and will be working over the next year to improve caching for authenticated requests. Community impact A key principle throughout this work has been to ensure responsible use of our infrastructure, whilst minimizing the impact on the community. Our communities rely on a broad ecosystem of bots and other tools to create and maintain the wikis, created by a dedicated group of technical volunteers. These limits are being put in place to protect our projects from high levels of abuse and ensure that we are able to better insulate our community infrastructure from high-volume commercial usage. They are necessary to ensure that developers are using the most appropriate channels, giving us the ability to prioritize the community and our human readers. Ideally, members of Wikimedia communities will not be affected by this change. However, it is possible that a small number of bots and other tools which operate at a very high rate may get rate limited. We ask you to follow the best practices and are here to help bot operators get the access they need. What we need you to do If you are a developer that uses Wikimedia APIs, we ask you to: - Read more about the new limits <https://www.mediawiki.org/wiki/Wikimedia_APIs/Rate_limits> and the updated Robot policy <https://wikitech.wikimedia.org/wiki/Robot_policy> - Update your tools/bots to follow the new best practices Should you require a higher rate of access, you are able to: - Request the bot flag <https://www.mediawiki.org/wiki/Manual:Bots> from your local wiki community - Consider running in Toolforge or another WMCS offering - Use Wikimedia Enterprise APIs <https://meta.wikimedia.org/wiki/Wikimedia_Enterprise#Access> for high-volume usage - Contact the WMF at bot-traffic(a)wikimedia.org Best Jonathan -- *Jonathan Tweed* (he/him) Senior Product Manager, Core Platform Wikimedia Foundation <https://wikimediafoundation.org/> _______________________________________________ Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud March 2026