We will be upgrading PAWS Kubernetes today at 2030UTC. User impacts should
be minimal, but you might see your notebook server stop and restart during
the change at some point.
Michael DiPietro
SRE Wikimedia Cloud Services
Quarry is currently running on python 3.5 on Debian Stretch. This is the
current version still running at quarry.wmflabs.org. A new version running
on python 3.7 on Debian Buster is now available at quarry.wmcloud.org. To
any interested party please test there and we will cut over the old domain
to the new buster systems if no problems are found in a few days.
Yesterday the hardworking developers at The Debian Project finalized the
latest version of Debian Linux, 'Bullseye' [0]. I've created a new
Bullseye base image for cloud-vps and it should now be accessible in all
projects.
There are likely to be bumps in the road with such a young release, but
the WMCS team is committed to supporting Bullseye so you should feel
confident adopting Bullseye for any new development. My cursory tests
look pretty good but if you encounter issues specific to Bullseye
and-cloud-vps please create a phabricator ticket or reply on the cloud
list[1].
-Andrew + the WMCS team
[0] https://www.debian.org/releases/bullseye/
[1] https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
Since there seems to be some error with sssd (LDAP and name services daemon) on the main Toolforge bastion, I am going to reboot it at 21:33 UTC today.
Sorry for the inconvenience.
Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
Tools admins will be upgrading Toolforge Kubernetes to version 1.19 on Monday July 26th at 1530UTC to catch up to the upstream release cycle. This should be mostly invisible to end users with the occasional pod restarting.
Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
We will be upgrading PAWS Kubernetes tomorrow at 1500UTC. User impacts should be minimal, but you might see your notebook server stop and restart during the change at some point. Calico (network overlay) may also be upgraded for both paws and tools, but previous upgrades have had no visible user impact at tall, so that should also be quiet and require no user action.
Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
A few weeks ago we rolled out a new service for Cloud VPS users:
OpenStack Trove, aka 'Database as a Service.'
Trove provides automatic orchestration of stand-alone database
instances. In brief, you tell Trove to create a database server with a
given size and backend, and it builds and manages the server and
provides you with ready-made access links. You can also manage databases
and users with Trove, or get a root prompt on the backend itself to
create users and databases.
We have only tested this a little bit, so I invite anyone with interest
to give this a try and let us know what works and what doesn't.
There's a longer blog post about this feature here:
https://techblog.wikimedia.org/2021/07/19/introducing-database-as-a-service…
And some slapdash user documentation here:
https://wikitech.wikimedia.org/wiki/Help:Adding_a_Database_to_a_Cloud_VPS_P…
Bugs and doc-patches are always welcome!
-Andrew + the WMCS team
Greetings!
Over the next two weeks our network staff will be adjusting and
restarting the eqiad network switches. This will affect every server and
service running on WMCS, both toolforge and cloud-vps.
We don't expect this to result in noticeable downtime, but any
connections that are active during the restarts will be interrupted.
It's also always possible that some unexpected side-effect will result
in a prolonged network outage.
One switch will be restarted at 15:00 UTC on July 20th, 22nd, 27th,
29th. The restart on the 27th is the most likely to affect cloud services.
To avoid worst-case scenarios the WMCS team will be failing over several
services before the restarts. Most of these changes won't be noticeable
to users but we'll notify in advance of impact if anything dramatic is
expected.
-Andrew
Hi there,
on Thurs July 22nd at 15:00 UTC (08:00 PDT / 11:00 EDT / 17:00 CEST) there is a
planned network maintenance that will affect the availability of the wiki
replica database service.
The expected operation window is of about 5 minutes long and it will affect any
wiki replicas users including Toolforge tools, PAWS, and any other Cloud VPS
project using them.
More information can be found on phabricator:
https://phabricator.wikimedia.org/T286614
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation