June 2024 - Cloud-announce - lists.wikimedia.org

Access to ToolsDB public dbs from Quarry and Superset
by Francesco Negri 02 Jul '24

02 Jul '24

Following several requests from users over the past eight years [0], we are finally enabling access to ToolsDB's "public" databases (the ones with a name ending with "_p") [1] from both Quarry [2] and Superset [3]. The data stored in those databases have always been accessible to every Toolforge user, but after this change they will become more broadly accessible, as Quarry can be accessed by anyone with a Wikimedia account, and saved queries in Quarry can be shared with public links that require no login at all. == This change is planned to go live on Monday, July 1st. == If you have any concerns or questions related to this change, please leave a comment in the Phabricator task or one of its subtasks. [0] Thanks to everyone for your patience and for keeping the task alive over the years! [0] https://phabricator.wikimedia.org/T151158 [1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Privileges_on_t… [2] https://meta.wikimedia.org/wiki/Research:Quarry [3] https://superset.wmcloud.org/ -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

1 1

Toolforge: introducing new security policy engine 2024-06-26 @ 08:30 UTZ
by Arturo Borrero Gonzalez 25 Jun '24

25 Jun '24

Hi there, tomorrow 2024-06-26 @ 08:30Z we will start enforcing new Kubernetes security rules in Toolforge [0]. We have taken measures to eliminate any user impact, but this being a potentially sensitive change, I wanted to send a heads up email. In a nut-shell, pod-related kubernetes resources, like Deployment or CronJob need to have a new set of security-related attributes correctly specified. This is because we are introducing Kyverno policies as a replacement of the deprecated PodSecurityPolicies (PSP) [1]. The new Kyverno policies have been deployed already, but are in 'Audit' mode. What we will be doing tomorrow is setting them to 'Enforce', which is the final step in this migration, before we can finally drop PSP [2]. Please, report [3] any disruption that you may see. regards. [0] https://phabricator.wikimedia.org/T368141 [1] https://phabricator.wikimedia.org/T279110 [2] https://phabricator.wikimedia.org/T364297 [3] https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication

1 0

Buster VM Deprecation Notice
by Seyram Komla Sapaty 24 Jun '24

24 Jun '24

Hello! You may have been notified about Buster VM deprecation on projects where you are not an admin. This was in error. Kindly ignore. Thanks! -- Seyram Komla Sapaty Developer Advocate Wikimedia Cloud Services

1 0

Cloudelastic migration scheduled for Thursday Jun 20, 1400 UTC
by Brian King 20 Jun '24

20 Jun '24

Hello Cloud users, Just wanted to give a heads-up for an upcoming maintenance for our cloudelastic hosts <https://wikitech.wikimedia.org/wiki/Help:CirrusSearch_elasticsearch_replicas>. We will be migrating them to a new load balancer starting Thursday, Jun 20 at 1400 UTC . The service will be down through the maintenance, which is expected to last between 1 and 2 hours. Fine details are available in this Phabricator task <https://phabricator.wikimedia.org/T367511> . If you have any questions about this, feel free to respond here or in IRC (see my nick below). Best, Brian King SRE, Data Platform/Search Platform Wikimedia Foundation IRC: inflatador

1 1

Internal Domain Name Support For Continuous Jobs
by Raymond Olisaemeka 20 Jun '24

20 Jun '24

Hi all! This is to let you know that Toolforge continuous jobs now support internal domain names! This means you can now make a request from any job type to a continuous job using the name of the continuous job directly, without having to know or keep track of its internal ephemeral IP address. To use it you need to specify `--port <number>` while creating your continuous job. Once done your continuous job can now be reached from other jobs with `https://<continuous-job-name>:<port>`. Note: This can only be configured for continuous jobs. Note: The name of the continuous job automatically becomes the internal domain name and there is currently no way to specify a custom name. Also, a reminder that you can find this and smaller user-facing updates about the Toolforge platform features here: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog Original task: https://phabricator.wikimedia.org/T348758 -- Ndibe Raymond Olisaemeka Software Engineer - Technical Engagement Wikimedia Foundation <https://wikimediafoundation.org/> <https://wikimediafoundation.org>

1 0

New 4th generation flavors in Cloud VPS
by Taavi Väänänen 17 Jun '24

17 Jun '24

Hi, I've added a new set of flavors on Cloud VPS.[0] The new flavors are almost identical to the existing `g3` flavors - the only difference is that instances on the new `g4` flavors get scheduled on hypervisors running more modern networking software.[1] When creating new instances, either by hand or via some automated means, please use the new flavors from now on. Using g3 flavors for new instances will be disabled in the coming days. We will be migrating existing VMs to the new flavors at some point in the near future. This will most likely involve briefly shutting down each VM as it's migrated from one network agent to the other. There will be a dedicated announcement for that once the timeline on that is more certain. As always, if your project needs some coordination before restarting things or this is otherwise a problem, feel free to contact us.[2] Migrating a g3 instance to a g4 flavor (or the other way around) without manual intervention would instead make that instance inaccessible. For that reason instance resizing has temporarily been disabled, if you need to resize an existing instance for whatever reason before it's been migrated please contact us. [0]: https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances#Instance_infor… [1]: https://phabricator.wikimedia.org/T364458 [2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation

1 1

No Friday plans? Spend your weekend replacing Buster VMs!
by Andrew Bogott 14 Jun '24

14 Jun '24

Cloud-vps users: There are now a mere two weeks remaining before Debian Buster ends its period of long term support. After June 30th, security upgrades will no longer be available for this release and VMs running Buster will become ever more risky and difficult to maintain. As of today there are still 143 Buster servers running in our cloud[0] -- some of them are probably yours! Please take some time to delete VMs that are no longer needed, and rebuild those that are still needed with a more modern release, ideally Debian Bookworm. There is a task for your project on phabricator[1] where you can update your progress. If you have vital VMs that you absolutely cannot rebuild by July 15th, please update the associated task with your plan and anticipated timeline. WMCS staff will start shutting down unacknowledged VMs in mid July in order to attract the attention of users who do not read email or follow phabricator. Buster's end of life has been a long time coming, and frequently announced. If you've been waiting for the right time to think about this, the time is now. Thank you! -Andrew + WMCS staff [0] https://os-deprecation.toolforge.org/ [1] https://phabricator.wikimedia.org/project/view/6373/

1 0

cloud-vps and toolforge outage in progress
by Andrew Bogott 11 Jun '24

11 Jun '24

There is as-of-yet undiagnosed issue with our storage system (ceph) which is causing serious failures throughout cloud-vps and toolforge. Multiple people are working on the issue, so watch this list for updates. Sorry for the downtime! -Andrew + wmcs team

1 1

Toolforge Redis timeout for idle connections
by Francesco Negri 03 Jun '24

03 Jun '24

Starting next week, May 27th, connections to Toolforge Redis [0] that remain idle for more than 10 minutes will be forcibly closed. [1] This change is in order to prevent a situation where too many connections are active [2] and Redis stops accepting new connections, causing an outage [3] for several tools. If your tool is using Toolforge Redis, we expect it to continue to work without issues, as most Redis clients automatically create a new connection to Redis in case an existing one is terminated. If you have a tool that relies on long-lived Redis connections and would be negatively impacted by this change, please let us know. [0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1029158 [1] https://phabricator.wikimedia.org/T363709 [2] https://wikitech.wikimedia.org/wiki/Incidents/2024-04-28_WMCS_Toolforge_Red… -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce June 2024