Cloud-announce April 2022

cloud-announce@lists.wikimedia.org

4 participants
7 discussions

Eliminating Debian Stretch in Cloud VPS
by Andrew Bogott 13 Oct '22

13 Oct '22

Debian Stretch's security support ends in mid 2022, and the Foundation's OS policy already discourages use of existing Stretch machines. That means that it's time for all project admins to start rebuilding your VMs with Bullseye (or, if you must, Buster.) Any webservices running in Kubernetes created in the last year or two are most likely using Buster images already, so there's no action needed for those. Older kubernetes jobs should be refreshed to use more modern images whenever possible. If you are still using the grid engine for webservices, we strongly encourage you to migrate your jobs to Kubernetes. For other grid uses, watch this space for future announcements about grid engine migration; we don't yet have a solution prepared for that. Details about the what and why for this process can be found here: https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation Here is the deprecation timeline: March 2021: Stretch VM creation disabled in most projects July 6, 2021: Active support of Stretch ends, Stretch moves into LTS <- You are Here -> January 1st, 2022: Stretch VM creation disabled in all projects, deprecation nagging begins in earnest. Stretch alternatives will be available for tool migration in Toolforge May 1, 2022: All active Stretch VMs will be shut down (but not deleted) by WMCS admins. This includes Toolforge grid exec nodes. June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will be deleted by WMCS admins

1 1

Reminder: All Debian Stretch VMs should be removed by May of 2022
by Andrew Bogott 29 Apr '22

29 Apr '22

There are still dozens of VMs in cloud-vps running Debian Stretch. All of these hosts will need to be deleted and replaced with VMs running either Buster or Bullseye in the next few months. Beginning in May we will begin to shut down Stretch instances. Please check this page for your projects, and take whatever steps are necessary to move off of Stretch: https://os-deprecation.toolforge.org/ Don't hesitate to reach out for help on IRC or mailing list if you need help with this migration. You may find Cinder volumes (https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_ins…) especially useful for transferring data between VMs. Note that we do NOT recommend in-place OS upgrades of VMs; it is almost always better to start with a fresh host and transfer workloads over. Details about the what and why for this process can be found here: https://wikitech.wikimedia.org/wiki/News/Stretch_deprecation Here is the remaining deprecation timeline: May 1, 2022: All active Stretch VMs will be shut down (but not deleted) by WMCS admins. June 30, 2022: LTS support for Debian Stretch ends, all Stretch VMs will be deleted by WMCS admins

1 1

[IMPORTANT] login.toolforge.org now points to Buster Grid Engine
by Seyram Komla Sapaty 29 Apr '22

29 Apr '22

Hello, As WMCS continue to work on deprecating Toolforge Stretch Grid Engine[0], we are today switching login.toolforge.org to point to Buster Grid Engine[1]. This is a minor change and it's not expected to disrupt current workflows. === What Does This Change Mean For Existing Users === This change does not directly impact any existing jobs. Users will now have access to the new toolforge-jobs framework and the latest versions of our runtime environments. Note that this change will result in ssh fingerprint warnings when you first connect. The new fingerprints can be found at https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.toolforge.o… and https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/dev.toolforge.org. [0]: https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation [1]: https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Time… -- Seyram Komla Sapaty Developer Advocate Wikimedia Cloud Services

1 0

Upgrading Openstack on 2022-05-02
by Vivian Rook 26 Apr '22

26 Apr '22

We will be running an Openstack upgrade from Victoria to Wallaby on 2022-05-02 Some networking interruptions will occur during the upgrade at 18:00 UTC https://zonestamp.toolforge.org/1651510817 -- *Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/>

1 0

Toolsdb outage this Tuesday, 2022-04-19 at 15:00 UTC
by Andrew Bogott 19 Apr '22

19 Apr '22

Next week (Tuesday, 2022-04-19 15:00 UTC) we will be upgrading the operating system that hosts the shared Toolsdb servers. This upgrade may take an hour or more, during which time the databases will not be available. This outage will be VERY DISRUPTIVE to many toolforge tools, as all database access will fail during the upgrade. Toolforge users may want to disable tools before the outage and/or check in to verify proper recovery after service is restored. There is likely to be a similar (or longer) outage in subsequent weeks as we also need to upgrade the database servers themselves. Toolsdb has grown to an ungainly size and can't be easily handled using standard rolling upgrade procedures; the WMCS team is in ongoing discussions about long-term solutions for this issue. In the meantime you can help us out by engaging in periodic cleanup of your database usage and dumping or dropping data that's no longer of use. -Andrew + the WMCS team

1 3

Brief service outage for some VMs
by Andrew Bogott 06 Apr '22

06 Apr '22

1 0

Network operations today 2022-04-06
by Arturo Borrero Gonzalez 06 Apr '22

06 Apr '22

Hi there, Today 2022-04-06 we're performing some network maintenance operations on Cloud VPS that could affect all cloud egress/ingress traffic, including Toolforge. The cuts, if noticeable, should last a few minutes at most. Some operations were also conducted yesterday (without this email notice), and some unexpected hiccups occurred. That's why the email today. regards. -- Arturo Borrero Gonzalez Site Reliability Engineer Wikimedia Cloud Services Wikimedia Foundation

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce April 2022