Debian Stretch's security support ends in mid 2022, and the Foundation's
OS policy already discourages use of existing Stretch machines. That
means that it's time for all project admins to start rebuilding your VMs
with Bullseye (or, if you must, Buster.)
Any webservices running in Kubernetes created in the last year or two
are most likely using Buster images already, so there's no action needed
for those. Older kubernetes jobs should be refreshed to use more modern
images whenever possible.
If you are still using the grid engine for webservices, we strongly
encourage you to migrate your jobs to Kubernetes. For other grid uses,
watch this space for future announcements about grid engine migration;
we don't yet have a solution prepared for that.
Details about the what and why for this process can be found here:
https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation
Here is the deprecation timeline:
March 2021: Stretch VM creation disabled in most projects
July 6, 2021: Active support of Stretch ends, Stretch moves into LTS
<- You are Here ->
January 1st, 2022: Stretch VM creation disabled in all projects,
deprecation nagging begins in earnest. Stretch alternatives will be
available for tool migration in Toolforge
May 1, 2022: All active Stretch VMs will be shut down (but not deleted)
by WMCS admins. This includes Toolforge grid exec nodes.
June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will
be deleted by WMCS admins
There are still dozens of VMs in cloud-vps running Debian Stretch. All
of these hosts will need to be deleted and replaced with VMs running
either Buster or Bullseye in the next few months. Beginning in May we
will begin to shut down Stretch instances.
Please check this page for your projects, and take whatever steps are
necessary to move off of Stretch:
https://os-deprecation.toolforge.org/
Don't hesitate to reach out for help on IRC or mailing list if you need
help with this migration. You may find Cinder volumes
(https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins…)
especially useful for transferring data between VMs. Note that we do NOT
recommend in-place OS upgrades of VMs; it is almost always better to
start with a fresh host and transfer workloads over.
Details about the what and why for this process can be found here:
https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation
Here is the remaining deprecation timeline:
May 1, 2022: All active Stretch VMs will be shut down (but not deleted)
by WMCS admins.
June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will
be deleted by WMCS admins
We will be running an Openstack upgrade from Victoria to Wallaby on
2022-05-02
Some networking interruptions will occur during the upgrade at 18:00 UTC
https://zonestamp.toolforge.org/1651510817
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
Next week (Tuesday, 2022-04-19 15:00 UTC) we will be upgrading the
operating system that hosts the shared Toolsdb servers. This upgrade may
take an hour or more, during which time the databases will not be available.
This outage will be VERY DISRUPTIVE to many toolforge tools, as all
database access will fail during the upgrade. Toolforge users may want
to disable tools before the outage and/or check in to verify proper
recovery after service is restored.
There is likely to be a similar (or longer) outage in subsequent weeks
as we also need to upgrade the database servers themselves. Toolsdb has
grown to an ungainly size and can't be easily handled using standard
rolling upgrade procedures; the WMCS team is in ongoing discussions
about long-term solutions for this issue. In the meantime you can help
us out by engaging in periodic cleanup of your database usage and
dumping or dropping data that's no longer of use.
-Andrew + the WMCS team
Hi there,
Today 2022-04-06 we're performing some network maintenance operations on
Cloud VPS that could affect all cloud egress/ingress traffic, including
Toolforge. The cuts, if noticeable, should last a few minutes at most.
Some operations were also conducted yesterday (without this email
notice), and some unexpected hiccups occurred. That's why the email today.
regards.
--
Arturo Borrero Gonzalez
Site Reliability Engineer
Wikimedia Cloud Services
Wikimedia Foundation