Cloud-announce June 2024

cloud-announce@lists.wikimedia.org

3 participants
5 discussions

New 4th generation flavors in Cloud VPS
by Taavi Väänänen 17 Jun '24

17 Jun '24

Hi, I've added a new set of flavors on Cloud VPS.[0] The new flavors are almost identical to the existing `g3` flavors - the only difference is that instances on the new `g4` flavors get scheduled on hypervisors running more modern networking software.[1] When creating new instances, either by hand or via some automated means, please use the new flavors from now on. Using g3 flavors for new instances will be disabled in the coming days. We will be migrating existing VMs to the new flavors at some point in the near future. This will most likely involve briefly shutting down each VM as it's migrated from one network agent to the other. There will be a dedicated announcement for that once the timeline on that is more certain. As always, if your project needs some coordination before restarting things or this is otherwise a problem, feel free to contact us.[2] Migrating a g3 instance to a g4 flavor (or the other way around) without manual intervention would instead make that instance inaccessible. For that reason instance resizing has temporarily been disabled, if you need to resize an existing instance for whatever reason before it's been migrated please contact us. [0]: https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances#Instance_infor… [1]: https://phabricator.wikimedia.org/T364458 [2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation

1 1

No Friday plans? Spend your weekend replacing Buster VMs!
by Andrew Bogott 14 Jun '24

14 Jun '24

Cloud-vps users: There are now a mere two weeks remaining before Debian Buster ends its period of long term support. After June 30th, security upgrades will no longer be available for this release and VMs running Buster will become ever more risky and difficult to maintain. As of today there are still 143 Buster servers running in our cloud[0] -- some of them are probably yours! Please take some time to delete VMs that are no longer needed, and rebuild those that are still needed with a more modern release, ideally Debian Bookworm. There is a task for your project on phabricator[1] where you can update your progress. If you have vital VMs that you absolutely cannot rebuild by July 15th, please update the associated task with your plan and anticipated timeline. WMCS staff will start shutting down unacknowledged VMs in mid July in order to attract the attention of users who do not read email or follow phabricator. Buster's end of life has been a long time coming, and frequently announced. If you've been waiting for the right time to think about this, the time is now. Thank you! -Andrew + WMCS staff [0] https://os-deprecation.toolforge.org/ [1] https://phabricator.wikimedia.org/project/view/6373/

1 0

Access to ToolsDB public dbs from Quarry and Superset
by Francesco Negri 13 Jun '24

13 Jun '24

Following several requests from users over the past eight years [0], we are finally enabling access to ToolsDB's "public" databases (the ones with a name ending with "_p") [1] from both Quarry [2] and Superset [3]. The data stored in those databases have always been accessible to every Toolforge user, but after this change they will become more broadly accessible, as Quarry can be accessed by anyone with a Wikimedia account, and saved queries in Quarry can be shared with public links that require no login at all. == This change is planned to go live on Monday, July 1st. == If you have any concerns or questions related to this change, please leave a comment in the Phabricator task or one of its subtasks. [0] Thanks to everyone for your patience and for keeping the task alive over the years! [0] https://phabricator.wikimedia.org/T151158 [1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Privileges_on_t… [2] https://meta.wikimedia.org/wiki/Research:Quarry [3] https://superset.wmcloud.org/ -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

1 0

cloud-vps and toolforge outage in progress
by Andrew Bogott 11 Jun '24

11 Jun '24

There is as-of-yet undiagnosed issue with our storage system (ceph) which is causing serious failures throughout cloud-vps and toolforge. Multiple people are working on the issue, so watch this list for updates. Sorry for the downtime! -Andrew + wmcs team

1 1

Toolforge Redis timeout for idle connections
by Francesco Negri 03 Jun '24

03 Jun '24

Starting next week, May 27th, connections to Toolforge Redis [0] that remain idle for more than 10 minutes will be forcibly closed. [1] This change is in order to prevent a situation where too many connections are active [2] and Redis stops accepting new connections, causing an outage [3] for several tools. If your tool is using Toolforge Redis, we expect it to continue to work without issues, as most Redis clients automatically create a new connection to Redis in case an existing one is terminated. If you have a tool that relies on long-lived Redis connections and would be negatively impacted by this change, please let us know. [0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1029158 [1] https://phabricator.wikimedia.org/T363709 [2] https://wikitech.wikimedia.org/wiki/Incidents/2024-04-28_WMCS_Toolforge_Red… -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce June 2024