Cloud-announce June 2021

cloud-announce@lists.wikimedia.org

2 participants
3 discussions

Webservice release changes coming this week
by Brooke Storm 01 Jul '21

01 Jul '21

Hello cloud users, We will be releasing webservice version 0.75 this week to Toolforge. Most of the changes should not be noticed (upgrading to python 3, preparing for grid system upgrades in the future), however, there is a change for some people who have started to use the service.template feature (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Webservice_templates <https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Webservice_templates>). This change https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/636… <https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/636…> affects the way webservice templates are found by adding more locations. It will automatically check certain code directories for your template besides $HOME, but it will throw an error if you have a symlink to one of these locations back to your tool’s home directory. The locations besides $HOME that this checks for a service.template are * ~/www/python/src * ~/www/js * ~/public_html If you are affected by that, then simply remove the symlink in $HOME, and it should work fine. If you are not symlinking a service.template back to your tool’s home directory, you should not notice any changes. The full changelog is at https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/700… <https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/700…> When the release is complete, I’ll send a followup email and record it in SAL. Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org

1 1

2021-07-01 scratch and maps NFS maintenance
by Brooke Storm 01 Jul '21

01 Jul '21

The NFS servers used for scratch and maps mounts (/data/project and /home in the maps project and /data/scratch in other projects) will be going offline for a short time tomorrow 2021-07-01 at around 1600 UTC to move the mounts to DRBD synced volumes. The current setup causes odd issues during failover including data loss and stale files left behind. The process taking place is one of those failovers so there may be some files that were previously deleted that need deleting again present and similar anomalies. I plan to reboot the maps project servers to make sure they have their mounts and processes restored as best as possible. The scratch mounts should be less impactful. If you use scratch, just be aware that it will go offline for a bit and will be back with some possible quirks. After that, the data should become far more stable and properly synced between the two systems. The process could start later than 1600 UTC if there are sync issues initially as I try to get as much of the data as possible transferred. More details here https://phabricator.wikimedia.org/T224747 <https://phabricator.wikimedia.org/T224747> Brooke Storm Staff SRE Wikimedia Cloud Services bstorm(a)wikimedia.org

1 2

Toolforge kubernetes upgrade Tuesday, 2021-06-29 at 17:00 UTC
by Andrew Bogott 29 Jun '21

29 Jun '21

Next Tuesday we will be upgrading Kubernetes on toolforge. As part of the upgrade we will need to restart all pods. This will produce a brief interruption in web services and other tools that use kubernetes. Assuming your services are able to survive a restart, no action should be needed on your part. I'll send a further email when the upgrade is finished. Special thanks to volunteer Taavi (aka Majavah) who has been essential in preparing for this upgrade and will be taking time out of his day to make sure the upgrade goes smoothly on Tuesday. -Andrew + the WMCS team

1 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce June 2021