- Cloud-admin - lists.wikimedia.org

Cloud VPS quota approval process
by Francesco Negri 04 Sep '25

04 Sep '25

The process for Cloud VPS quota requests used to state that "Requests that represent an increase of more than double the quota or more than 300GB of storage should also be reviewed at the weekly meeting." I don't think this was strictly followed in the past, and it could block even relatively small requests for up to one week. I boldly edited the Clinic Duties wiki page [0] to suggest those requests should have two "+1"s instead. If you disagree with this change, please let me know and we can discuss it during the next team meeting. [0] https://wikitech.wikimedia.org/w/index.php?title=Wikimedia_Cloud_Services_t… -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

2 1

[2025-08-01 08:00:00UTC] Toolsbeta k8s upgrade to 1.30
by David Caro 28 Aug '25

28 Aug '25

Hi! Just a heads up, I'm planning on upgrading toolsbeta k8s to 1.30[1] on monday at around 08:00 UTC time. I'll update live on irc during the upgrade, note that if you want to deploy/test/run anything there it might stop working momentarily. [1] https://phabricator.wikimedia.org/T402377 -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment."

1 0

New Toolsbeta bastion
by Taavi Väänänen 28 Aug '25

28 Aug '25

There is a new Toolsbeta bastion running trixie, toolsbeta-bastion-7. I've updated the login.toolsbeta.org service alias to point to it, the SSH key fingerprint can be verified from [0]. [0]: https://toolsbeta-static.wmcloud.org/admin/fingerprints/ If you use some other name than `login.toolsbeta.org` to connect, please update your config/memory/bash history. There's a variety of old names under other domains that I plan to remove when removing the old bastion instead of updating them. Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation

1 0

Cloud VPS and IPv6
by Arturo Borrero Gonzalez 24 Apr '25

24 Apr '25

Hi there, for the record, yesterday we enabled the Dualstack IPv6 network on Cloud VPS [0]. We will make it "official" [1] very soon. regards. [0] https://phabricator.wikimedia.org/T380174 [1] https://wikitech.wikimedia.org/wiki/News/2025_Cloud_VPS_VXLAN_IPv6_migration

2 1

Toolforge development should not be happening behind closed doors (again)
by Taavi Väänänen 11 Apr '25

11 Apr '25

Apologies in advance for a possibly ranty email. I'm again seeing some major Toolforge developments happening with major decisions being made in private. A few examples: * Infrastructure IaC/OpenTofu migration (with an intern assigned to it) popping out of nowhere to implementation tasks https://phabricator.wikimedia.org/T390056 with no public/formal discussion whether we want to replace the current tooling with that. * UI project starting an "investigation" task (https://phabricator.wikimedia.org/T383146) with a clear bias towards a specific solution, based on unspecific "previous discussions", which were then hidden from the description when I asked for documents relating those to be made public. * The Toolforge CLI consolidation project apparently has switched implementation languages compared to the last decision request, apparently based on "a team decision" that was apparently made "a few months" ago but never communicated anywhere as far as I can tell: https://phabricator.wikimedia.org/T356262#10722410 First, could we please again stop doing major decisions in private? Second, is there something in our processes or workflows that we could improve so that things wouldn't always immediately start going more hidden and private every time I stop paying close attention to this? Taavi

4 3

Quotas support in tofu-infra
by Arturo Borrero Gonzalez 09 Apr '25

09 Apr '25

Hi there, the tofu-infra project gained support for managing quotas, see https://phabricator.wikimedia.org/T371391 Openstack quotas can now be established in several ways: * traditionally via the wmcs-openstack CLI * using the cookbook to manage quotas * using tofu-infra If a quota is set via tofu-infra, it will override all the others, as far as I understand how they work. If a quota is not set via tofu-infra, then whatever quota was already established should be in action. I have not back-filled, nor have plans to do so, previous quotas into tofu-infra. regards.

1 0

Network maintenance operation on 2025-03-27 12:30 UTC
by Arturo Borrero Gonzalez 27 Mar '25

27 Mar '25

Hello, there will be a network maintenance operation happening on Thursday 2025-03-27 12:30 UTC that may impact all Wikimedia Cloud Services, including: * Cloud VPS * Toolforge * PAWS * and any others The planned operation window will last for 1h, from 12:30 UTC to 13:30 UTC. We have prepared the required changes and we are expecting no outages. But the operation is sensitive and we would like you to be informed of potential service disruptions as a result of the operations. If you want more technical details, we are using a Phabricator ticket: https://phabricator.wikimedia.org/T389958 regards.

2 2

opentofu users: please update network name!
by Arturo Borrero Gonzalez 26 Mar '25

26 Mar '25

Hello, If you have a Cloud VPS project and use opentofu to manage virtual resources, this message is for your -- otherwise feel free to ignore. In particular, Toolforge users are free to ignore this message. The main Cloud VPS virtual network resource name is changing: * from: lan-flat-cloudinstances2b * to: VLAN/legacy If your opentofu / terraform code mentions this network name, you will need to update it. See also Phabricator ticket: https://phabricator.wikimedia.org/T389942 regards.

1 0

Get your files off of cloudcontrol1005
by Andrew Bogott 04 Dec '24

04 Dec '24

Cloudcontrol1005 is due for a refresh, as per T379373 There's rather a lot in /home on that host -- please have a look at your home dir there and rescue the files that you need. For things that need to live on a cloudcontrol, cloudcontrol1007 will the the longest-lived member of that cluster. I will probably not send another of these emails, so act now before your files are lost! root@cloudcontrol1005:/home# du -h -d1 . | grep -v 8.0K | grep -v 12K | grep -v 16K | grep -v 20K # exclude dirs with puppetized .files 96K ./klausman 28K ./legoktm 24K ./volans 947M ./andrew 328K ./ori 32K ./cdanis 148K ./otto 28K ./swfrench 24K ./ryankemper 336K ./sstefanova 168K ./aborrero 72K ./raymond-ndibe 1.2M ./fnegri 240K ./jhathaway 32K ./oblivian 24K ./btullis 140K ./kormat 552K ./akosiaris 36K ./jynus 28K ./samtar 1.2M ./jayme 172K ./bd808 1.6G ./rook

1 0

regarding tofu-infra, workflows, scope, roadmaps and use cases
by Arturo Borrero Gonzalez 21 Nov '24

21 Nov '24

Hi, the WMCS team just had a meeting dedicated to tofu-infra [0], workflows, scope, roadmaps and use cases. This is a summary. * The current state, and short/mid term roadmaps have been shared, which includes refactoring the tofu-infra repo, and adding support for more stuff, for example quotas. * It was mentioned that user assignment is maybe not very well tracked in tofu-infra, because that list can be modified at any given time by end users. * We kind of agreed that the scope of tofu-infra repository [1] is to track state for admin-controlled openstack resources. User-managed resources are explicitly out of the scope of tofu-infra. Tracking Toolforge resources (k8s VMs and such) are also out of the scope this tofu-infra repository. * There is some controversy regarding tracking projects resources. The project definition itself. There is a real concern with project deletion, because it will leave orphan resources. * Because the above point, we have been discussing _not_ tracking project definitions in tofu-infra at all. And automate project deletion via a cookbook that takes care of removing potentially orphaned resources. * Obviously not tracking projects comes with the downside of... well, not having gitops for projects definitions, which is something we kind of want to have. * We have been discussing potential solutions, including: ** extending wmfkeystonehook to prevent deleting projects if they have resources ** expand our suite of 'leak detector' scripts to report orphaned resources ** having some kind of background daemon automatically cleaning up orphaned resources * A strong decision on what to do with project definitions has not been made, which implies we should keep the current status quo (they are tracked in tofu-infra). * We have been discussing ideas and options for integrating tofu-infra with the cookbooks, with a variety of opinions on whether that is convenient or not, or if tofu-infra is ready for building automation on top of it or not, etc. * It was mentioned the use case, or desire, to enable our clinic-duty workflows for non-technical, non-SRE folks in the organization. While the use case is felt right, some bits should be clarified, because these folks may not have the required access level after all, for example to use cookbooks. Finally, it was agreed that further discussions should happen on this topic soon. regards. [0] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/OpenTofu [1] https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin