Due to unavoidable network switch maintenance, some WMCS services
will be offline briefly tomorrow. The downtime will last for 20-30
minutes and take place sometime between 14:00 and 16:00 UTC.
Here is what to expect during the downtime:
* *Toolsdb will be unavailable and all queries will fail*
* Some of the wiki replica databases may be unavailable
* Some DNS servers will be offline; some services may fail to resolve
hosts, depending on their fallback logic
We anticipate a graceful recovery from this outage, but NFS is fickle so
we may need to reboot some or all VMs after the outage.
Sorry in advance for any inconvenience or upset emails that result from
- Andrew + the WMCS team
We will be upgrading the Toolforge Kubernetes cluster next Monday
(2023-04-03) starting at around 10:00 UTC.
The expected impact is that tools running on the Kubernetes cluster will
get restarted a couple of times over the course of the few hours it
takes for us to upgrade the entire cluster. The ability to manage tools
will remain operational.
Since the version we're upgrading to (1.22) removes a bunch of
deprecated Kubernetes APIs, tools that use kubectl and raw Kubernetes
resources directly may want to check that they're on the latest
available versions. The vast majority of tools that are only using the
Jobs framework and/or the webservice command are not affected by these
PAWS will be switching k8s clusters to get to the latest k8s that openstack
currently supports (1.23). This should occur on 2023-03-20 around 13:00
UTC. Anything that was running at the time on the current (old) cluster
will need restarted.
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
I am in the process of standardizing the role names in WMCS cloud-vps
to conform with upstream conventions. That requires me to rename two
existing user roles, 'user' and 'projectadmin':
- The role previously called 'user' will now be called 'reader'
- The role previously called 'projectadmin' will now be called 'member'
Despite the (IMO) less obvious names, a 'reader' can still log into
project VMs, and a 'member' can still create and delete VMs. Taavi has
thoughtfully upgraded the documentation about what roles can do what;
the complete docs can be found at
This renaming is phase one; phase two will involve switching to the
default upstream access rules for these two new roles.
Right now the old and new roles are co-existing in our system, but soon
I will entirely delete the old 'user' and 'projectadmin' roles. In the
meantime, please let me know if you find stray references to the old
role names, or if you find yourself unable to perform Horizon actions
that you were previously able to do. Or, more seriously, able to do
things that you were not previously able to do!
Sorry for any inconvenience caused!
 Our OpenStack deployment has a very long history; it is older than
most deployments. That means that many conventions established in our
cloud now differ from the consensus standards created among newer
clouds. Periodically I try to update our cloud to conform to these new
standards; it reduces tech debt and also increases the chances that
official OpenStack documentation will be useful to our users.
 There is one edge case in Horizon that may require you to switch
projects in order to refresh the role permissions.
Today 2023-03-06, in a few minutes, we will restart the Toolforge internal
network, A brief interruption of network communications is expected during the
This is because we're re-deploying calico to the kubernetes cluster .
No action required on your side.
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
As part of the ToolsDB migration work , in about 1 hour from now I
will stop ToolsDB for a very short time (I expect the downtime to last
approximately 2 minutes).
You can follow along and report any issues in the #wikimedia-cloud IRC channel.
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team