Debian Stretch's security support ends in mid 2022, and the Foundation's
OS policy already discourages use of existing Stretch machines. That
means that it's time for all project admins to start rebuilding your VMs
with Bullseye (or, if you must, Buster.)
Any webservices running in Kubernetes created in the last year or two
are most likely using Buster images already, so there's no action needed
for those. Older kubernetes jobs should be refreshed to use more modern
images whenever possible.
If you are still using the grid engine for webservices, we strongly
encourage you to migrate your jobs to Kubernetes. For other grid uses,
watch this space for future announcements about grid engine migration;
we don't yet have a solution prepared for that.
Details about the what and why for this process can be found here:
https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation
Here is the deprecation timeline:
March 2021: Stretch VM creation disabled in most projects
July 6, 2021: Active support of Stretch ends, Stretch moves into LTS
<- You are Here ->
January 1st, 2022: Stretch VM creation disabled in all projects,
deprecation nagging begins in earnest. Stretch alternatives will be
available for tool migration in Toolforge
May 1, 2022: All active Stretch VMs will be shut down (but not deleted)
by WMCS admins. This includes Toolforge grid exec nodes.
June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will
be deleted by WMCS admins
An upgrade to JupyterHub is going out on 2021-10-26. There will be a new
singleuser container as a result. Currently running containers may need
restarting.
Thank you,
Michael DiPietro
Cloud Services SRE
If you were running a Toolforge web tool in Kubernetes before the toollabs-webservice label changes were deployed on 2021-09-29 (https://sal.toolforge.org/tools?d=2021-09-29 <https://sal.toolforge.org/tools?d=2021-09-29>). You may need to run `webservice stop && webservice start` in order to ensure your replica sets have correct label expectations on them going forward. Otherwise you may find confusing states may happen when running webservice restart and similar commands.
When I backfilled the new labels, I missed that you cannot change the label matching rules in a deployment retroactively. I apologize for any inconvenience.
In summary: If you haven’t run a webservice stop since 2021-09-29 on your Kubernetes web service, it would be a good idea to stop and start your webservice now to prevent any confusing behavior from webservice in the future.
--
Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
Next Tuesday we will be upgrading Kubernetes on toolforge. As part of the
upgrade we will need to restart all pods. This will produce a brief
interruption in web services and other tools that use kubernetes. Assuming
your services are able to survive a restart, no action should be needed on
your part.
Thank you,
Michael + the WMCS team