Debian Stretch's security support ends in mid 2022, and the Foundation's
OS policy already discourages use of existing Stretch machines. That
means that it's time for all project admins to start rebuilding your VMs
with Bullseye (or, if you must, Buster.)
Any webservices running in Kubernetes created in the last year or two
are most likely using Buster images already, so there's no action needed
for those. Older kubernetes jobs should be refreshed to use more modern
images whenever possible.
If you are still using the grid engine for webservices, we strongly
encourage you to migrate your jobs to Kubernetes. For other grid uses,
watch this space for future announcements about grid engine migration;
we don't yet have a solution prepared for that.
Details about the what and why for this process can be found here:
https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation
Here is the deprecation timeline:
March 2021: Stretch VM creation disabled in most projects
July 6, 2021: Active support of Stretch ends, Stretch moves into LTS
<- You are Here ->
January 1st, 2022: Stretch VM creation disabled in all projects,
deprecation nagging begins in earnest. Stretch alternatives will be
available for tool migration in Toolforge
May 1, 2022: All active Stretch VMs will be shut down (but not deleted)
by WMCS admins. This includes Toolforge grid exec nodes.
June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will
be deleted by WMCS admins
There are still dozens of VMs in cloud-vps running Debian Stretch. All
of these hosts will need to be deleted and replaced with VMs running
either Buster or Bullseye in the next few months. Beginning in May we
will begin to shut down Stretch instances.
Please check this page for your projects, and take whatever steps are
necessary to move off of Stretch:
https://os-deprecation.toolforge.org/
Don't hesitate to reach out for help on IRC or mailing list if you need
help with this migration. You may find Cinder volumes
(https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins…)
especially useful for transferring data between VMs. Note that we do NOT
recommend in-place OS upgrades of VMs; it is almost always better to
start with a fresh host and transfer workloads over.
Details about the what and why for this process can be found here:
https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation
Here is the remaining deprecation timeline:
May 1, 2022: All active Stretch VMs will be shut down (but not deleted)
by WMCS admins.
June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will
be deleted by WMCS admins
PAWS upgrading to pywikibot 7.0.0 there are some breaking changes:
Support of Python 3.5.0 - 3.5.2 has been dropped (T286867)
generate_user_files.py, generate_user_files.py, shell.py and version.py
were moved to pywikibot/scripts and must be used with pwb wrapper script
With some more in the Code cleanups section of the changelog:
https://doc.wikimedia.org/pywikibot/stable/changelog.html
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
In about an hour (at 15:00 UTC today) we'll be upgrading the networking
servers for cloud-vps. This may cause brief networking interruptions for
both cloud-vps and toolforge.
No action should be needed on your part.
-Andrew + the WMCS team
Hello, all!
This is a follow-up on our earlier announcement[0] of the above.
Thanks to those who have already migrated their tool(s) from Debian Stretch
grid or are
in the process of doing this.
At the start of this process, there were 867 tools running on Stretch grid.
The current number is 821.
=== Recap ===
We are migrating away from Debian Stretch[1] to Debian Buster for all of
Toolforge servers,
and the most affected piece is the Grid Engine backend in particular.
We need to shut down all Stretch hosts before the end of support date to
ensure that
Toolforge remains a secure platform. This migration will take several
months because many people still use the Stretch hosts and our users
are working on tools in their spare time.
== What should I do? ==
You should migrate your Toolforge tool to a newer environment.
You have two options:
* migrate from Toolforge Stretch Grid Engine to Toolforge Kubernetes[2].
* migrate from Toolforge Stretch Grid Engine to Toolforge Buster Grid
Engine.[3]
== Timeline ==
* 2022-02-15: Availability of Debian Buster grid announced to community -
DONE
* 2022-03-21: Weekly reminders via email to tool maintainers for tools
still running on Stretch - IN PROGRESS
* Week of 2022-04-21:
** Daily reminders via email to tool maintainers for tools still running on
Stretch
** Switch login.toolforge.org to point to Buster bastion
* Week of 2022-05-02: Evaluate migration status and formulate plan for
final shutdown of Stretch grid
* Week of 2022-05-21: Shutdown Stretch grid
We thank all of you for your support during this migration process.
You can always reach out via any of our communication channels[4]
[0]
https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.…
[1] https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation
[2]
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engi…
[3]
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move…
[4]
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Commun…
Thanks.
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
In a few hours we'll be replacing the existing cloud-vps bastions with
new systems running Debian Bullseye.
Because this is a DNS change, existing bastion sessions should not be
interrupted. New connections will produce fingerprint warnings that will
require you to update your .ssh/known_hosts. Here are the fingerprints
for the new systems:
primary.bastion.wmcloud.org, eqiad1.bastion.wmcloud.org,
bastion.wmcloud.org:
ED25519 key fingerprint is
SHA256:QlZONtScYR4O5jGnrmKRhWVF9lJE+aReENpHXqeOL/4
secondary.bastion.wmcloud.org:
ED25519 key fingerprint is
SHA256:tRgnLMmISSuByzzeX8yXWcdFKjZad8Hdy6Y7E6jgaGI
-Andrew + the WMCS team