- Cloud-announce - lists.wikimedia.org

Upgrade to paws JupyterHub 2021-10-26
by Michael DiPietro 26 Oct '21

26 Oct '21

An upgrade to JupyterHub is going out on 2021-10-26. There will be a new singleuser container as a result. Currently running containers may need restarting. Thank you, Michael DiPietro Cloud Services SRE

1 0

Toolforge: need to stop webservice and start it again following changes to labels
by Brooke Storm 14 Oct '21

14 Oct '21

If you were running a Toolforge web tool in Kubernetes before the toollabs-webservice label changes were deployed on 2021-09-29 (https://sal.toolforge.org/tools?d=2021-09-29 <https://sal.toolforge.org/tools?d=2021-09-29>). You may need to run `webservice stop && webservice start` in order to ensure your replica sets have correct label expectations on them going forward. Otherwise you may find confusing states may happen when running webservice restart and similar commands. When I backfilled the new labels, I missed that you cannot change the label matching rules in a deployment retroactively. I apologize for any inconvenience. In summary: If you haven’t run a webservice stop since 2021-09-29 on your Kubernetes web service, it would be a good idea to stop and start your webservice now to prevent any confusing behavior from webservice in the future. -- Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org

1 0

Toolforge kubernetes upgrade Tuesday, 2021-10-12 at 13:00 UTC
by Michael DiPietro 08 Oct '21

08 Oct '21

Next Tuesday we will be upgrading Kubernetes on toolforge. As part of the upgrade we will need to restart all pods. This will produce a brief interruption in web services and other tools that use kubernetes. Assuming your services are able to survive a restart, no action should be needed on your part. Thank you, Michael + the WMCS team

1 0

[Toolforge] Some tools broken by upstream Let's Encrypt certificate changes
by Bryan Davis 01 Oct '21

01 Oct '21

TL;DR: * Let's Encrypt [0] TLS certificates are "signed" by "root" certificates to create a chain of trust * The oldest "root" signing certificate for LE certs (DST Root CA X3) expired on 2021-09-30 [1] * Deprecated Toolforge Kubernetes containers only knew this root certificate and not the newer root certificate (ISRG Root X1) * Update your tool to a newer container to fix We are starting to hear reports of tools that suddenly stopped working on 2021-09-30. The common issue is accessing the APIs for Wikimedia wikis. The Wikimedia wikis use multiple TLS certificates issued by different providers for redundancy and protection against a problem with a single certificate provider. One of the certificate providers that we use is Let's Encrypt (LE) [0]. LE certificates are themselves signed by multiple "root" certificates to create a chain of trust that your web browser or other TLS verifying software can trust. The oldest root certificate (named "DST Root CA X3") used to sign the LE certificates expired on 2021-09-30 [1]. Very old operating systems and some compiled software do not have the newer root certificate (named "ISRG Root X1") in their trusted certificate collection. These systems are now rejecting LE certificates. In Toolforge, we think that this mainly affects tools running on the Kubernetes cluster inside Debian Jessie based containers. Specifically the "php5.6", "python", "python2", and "ruby2" containers are expected to have issues with the LE certificate expiration based on what we have found so far. Recommended replacement containers are "php7.4", "python3.9", and "ruby25". We also have reports of `mono` on the bastions + grid engine failing. We do not yet have a fix for this. It will require us to compile and install a newer version of mono for everyone who is using it. Interested folks can follow progress of our infrastructure updates in response to this issue at T291387 [3]. [0]: https://letsencrypt.org/ [1]: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ [2]: https://phabricator.wikimedia.org/T291387 Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

abuse_filter_log.afl_filter column will be dropped from the wiki replicas views on 2021-10-06
by Michael DiPietro 29 Sep '21

29 Sep '21

Due to Mediawiki schema changes from: https://phabricator.wikimedia.org/T291719 the abuse_filter_log.afl_filter column will be dropped from the wiki replicas views on 2021/10/06. We apologize for any inconvenience. Please update queries accordingly.

1 0

Update in time for PAWS Kubernetes Upgrade
by Michael DiPietro 07 Sep '21

07 Sep '21

We will be upgrading PAWS Kubernetes 2021/09/07 at 1500UTC. User impacts should be minimal. but you might see your notebook server stop and restart during the change at some point. Michael DiPietro SRE Wikimedia Cloud Services

1 1

Upgrading paws k8s
by Michael DiPietro 01 Sep '21

01 Sep '21

We will be upgrading PAWS Kubernetes today at 2030UTC. User impacts should be minimal, but you might see your notebook server stop and restart during the change at some point. Michael DiPietro SRE Wikimedia Cloud Services

1 0

quarry upgrade to Buster
by Michael DiPietro 31 Aug '21

31 Aug '21

Quarry is currently running on python 3.5 on Debian Stretch. This is the current version still running at quarry.wmflabs.org. A new version running on python 3.7 on Debian Buster is now available at quarry.wmcloud.org. To any interested party please test there and we will cut over the old domain to the new buster systems if no problems are found in a few days.

1 0

New Kubernetes runtime images available on Toolforge
by Taavi Väänänen 17 Aug '21

17 Aug '21

Hi! I've added several new Kubernetes container images[0] based on the new Debian version 11 "bullseye"[1]. The new images are: * Java 17 * Node.js 12 * PHP 7.4 * Python 3.9 * Ruby 2.7 If you see any issues with those new images, please file a new Phabricator task as a subtask of T284590[2]. [0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Container_ima… [1]: https://www.debian.org/releases/bullseye/ [2]: https://phabricator.wikimedia.org/T284590 -- Taavi Väänänen (User:Majavah)

1 0

Debian 11.0 'Bullseye' now available on cloud-vps
by Andrew Bogott 15 Aug '21

15 Aug '21

Yesterday the hardworking developers at The Debian Project finalized the latest version of Debian Linux, 'Bullseye' [0]. I've created a new Bullseye base image for cloud-vps and it should now be accessible in all projects. There are likely to be bumps in the road with such a young release, but the WMCS team is committed to supporting Bullseye so you should feel confident adopting Bullseye for any new development. My cursory tests look pretty good but if you encounter issues specific to Bullseye and-cloud-vps please create a phabricator ticket or reply on the cloud list[1]. -Andrew + the WMCS team [0] https://www.debian.org/releases/bullseye/ [1] https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce