- Cloud-announce - lists.wikimedia.org

Internal Domain Name Support For Continuous Jobs
by Raymond Olisaemeka 20 Jun '24

20 Jun '24

Hi all! This is to let you know that Toolforge continuous jobs now support internal domain names! This means you can now make a request from any job type to a continuous job using the name of the continuous job directly, without having to know or keep track of its internal ephemeral IP address. To use it you need to specify `--port <number>` while creating your continuous job. Once done your continuous job can now be reached from other jobs with `https://<continuous-job-name>:<port>`. Note: This can only be configured for continuous jobs. Note: The name of the continuous job automatically becomes the internal domain name and there is currently no way to specify a custom name. Also, a reminder that you can find this and smaller user-facing updates about the Toolforge platform features here: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog Original task: https://phabricator.wikimedia.org/T348758 -- Ndibe Raymond Olisaemeka Software Engineer - Technical Engagement Wikimedia Foundation <https://wikimediafoundation.org/> <https://wikimediafoundation.org>

1 0

Cloudelastic migration scheduled for Thursday Jun 20, 1400 UTC
by Brian King 18 Jun '24

18 Jun '24

Hello Cloud users, Just wanted to give a heads-up for an upcoming maintenance for our cloudelastic hosts <https://wikitech.wikimedia.org/wiki/Help:CirrusSearch_elasticsearch_replicas>. We will be migrating them to a new load balancer starting Thursday, Jun 20 at 1400 UTC . The service will be down through the maintenance, which is expected to last between 1 and 2 hours. Fine details are available in this Phabricator task <https://phabricator.wikimedia.org/T367511> . If you have any questions about this, feel free to respond here or in IRC (see my nick below). Best, Brian King SRE, Data Platform/Search Platform Wikimedia Foundation IRC: inflatador

1 0

New 4th generation flavors in Cloud VPS
by Taavi Väänänen 17 Jun '24

17 Jun '24

Hi, I've added a new set of flavors on Cloud VPS.[0] The new flavors are almost identical to the existing `g3` flavors - the only difference is that instances on the new `g4` flavors get scheduled on hypervisors running more modern networking software.[1] When creating new instances, either by hand or via some automated means, please use the new flavors from now on. Using g3 flavors for new instances will be disabled in the coming days. We will be migrating existing VMs to the new flavors at some point in the near future. This will most likely involve briefly shutting down each VM as it's migrated from one network agent to the other. There will be a dedicated announcement for that once the timeline on that is more certain. As always, if your project needs some coordination before restarting things or this is otherwise a problem, feel free to contact us.[2] Migrating a g3 instance to a g4 flavor (or the other way around) without manual intervention would instead make that instance inaccessible. For that reason instance resizing has temporarily been disabled, if you need to resize an existing instance for whatever reason before it's been migrated please contact us. [0]: https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances#Instance_infor… [1]: https://phabricator.wikimedia.org/T364458 [2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation

1 1

No Friday plans? Spend your weekend replacing Buster VMs!
by Andrew Bogott 14 Jun '24

14 Jun '24

Cloud-vps users: There are now a mere two weeks remaining before Debian Buster ends its period of long term support. After June 30th, security upgrades will no longer be available for this release and VMs running Buster will become ever more risky and difficult to maintain. As of today there are still 143 Buster servers running in our cloud[0] -- some of them are probably yours! Please take some time to delete VMs that are no longer needed, and rebuild those that are still needed with a more modern release, ideally Debian Bookworm. There is a task for your project on phabricator[1] where you can update your progress. If you have vital VMs that you absolutely cannot rebuild by July 15th, please update the associated task with your plan and anticipated timeline. WMCS staff will start shutting down unacknowledged VMs in mid July in order to attract the attention of users who do not read email or follow phabricator. Buster's end of life has been a long time coming, and frequently announced. If you've been waiting for the right time to think about this, the time is now. Thank you! -Andrew + WMCS staff [0] https://os-deprecation.toolforge.org/ [1] https://phabricator.wikimedia.org/project/view/6373/

1 0

Access to ToolsDB public dbs from Quarry and Superset
by Francesco Negri 13 Jun '24

13 Jun '24

Following several requests from users over the past eight years [0], we are finally enabling access to ToolsDB's "public" databases (the ones with a name ending with "_p") [1] from both Quarry [2] and Superset [3]. The data stored in those databases have always been accessible to every Toolforge user, but after this change they will become more broadly accessible, as Quarry can be accessed by anyone with a Wikimedia account, and saved queries in Quarry can be shared with public links that require no login at all. == This change is planned to go live on Monday, July 1st. == If you have any concerns or questions related to this change, please leave a comment in the Phabricator task or one of its subtasks. [0] Thanks to everyone for your patience and for keeping the task alive over the years! [0] https://phabricator.wikimedia.org/T151158 [1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Privileges_on_t… [2] https://meta.wikimedia.org/wiki/Research:Quarry [3] https://superset.wmcloud.org/ -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

1 0

cloud-vps and toolforge outage in progress
by Andrew Bogott 11 Jun '24

11 Jun '24

There is as-of-yet undiagnosed issue with our storage system (ceph) which is causing serious failures throughout cloud-vps and toolforge. Multiple people are working on the issue, so watch this list for updates. Sorry for the downtime! -Andrew + wmcs team

1 1

Toolforge Redis timeout for idle connections
by Francesco Negri 03 Jun '24

03 Jun '24

Starting next week, May 27th, connections to Toolforge Redis [0] that remain idle for more than 10 minutes will be forcibly closed. [1] This change is in order to prevent a situation where too many connections are active [2] and Redis stops accepting new connections, causing an outage [3] for several tools. If your tool is using Toolforge Redis, we expect it to continue to work without issues, as most Redis clients automatically create a new connection to Redis in case an existing one is terminated. If you have a tool that relies on long-lived Redis connections and would be negatively impacted by this change, please let us know. [0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Redis_for_Toolforge [1] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1029158 [1] https://phabricator.wikimedia.org/T363709 [2] https://wikitech.wikimedia.org/wiki/Incidents/2024-04-28_WMCS_Toolforge_Red… -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

1 1

Cloud-vps users: please claim your projects
by Andrew Bogott 24 May '24

24 May '24

If you are a user of cloud-vps, please visit the following page and mark any projects that are still in use: https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2024_Purge There are currently 95 unmarked projects on that page; any that remain unclaimed will be deemed abandoned and subject to shutdown in June. I'm attaching the names of those 95 projects at the end of this email. If you see something on that list that you know to be important, please make a note on the wiki page (even if you are not an admin) so we know that somebody cares. Thanks! Unclaimed projects as of 2024-05-24: adiutor automation-framework baglama2 canasta2_test capacity-exchange citefix clouddb-services codereview commons-corruption-checker download duct dump-references-processor entity-detection etytree extdist foundationmemory gitlab-test glamwikidashboard gratitude hoiscript huggle huwiki-dev idm-dev image-suggestion-api imagebulk impactvisualizer integration ipoidopensearch isa k8splay language ldap-dev linkwatcher lta-tracker lutz machine-learning maps-experiments math matrix media-streaming mediawiki-vagrant mix-n-match mwcli mwoffliner mwv-apt onfire openvas packaging petscan pixel policy-test-project puppet-dev rcm reading-web-staging schematreerecommender search security-tools service shiny-r signwriting spi-tools sre-sandbox sso superset text-to-speech teyora tf-infra-test traffic video videocuttool visualeditor vuessr wcdo webperf wikibase-registry wikicite wikicommunityhealth wikidata-dev wikidata-history-query-service wikidata-query wikidocumentaries wikidumpparse wikifunctions wikilabels wikimania-mautic wikipathways wikispeech wikitextexp wildcat wm-bot wmcs-uptime wmcz-stats wmde-dashboards wmdeanalytics wmf-dumps-playground

1 0

Cloud VPS network maintenance on 2024-05-21
by Taavi Väänänen 14 May '24

14 May '24

We will be performing maintenance on the Cloud VPS network next Tuesday (2024-05-21) starting at around 14:00 UTC. During this window we will be replacing some software on the Cloud VPS network router.[0] During the maintenance window there will be a brief period during which Cloud VPS and any services hosted there (including Toolforge and PAWS) will not have any external network connectivity. Based on tests done in our staging environment the full outage should last for less than a minute assuming there are no unexpected issues. There is no action required on your side, unless your tools are not resilient to unexpected network outages - in that case you may need to manually restart those tools after the maintenance is complete. [0]: https://phabricator.wikimedia.org/T364459 Taavi (+ the rest of the WMCS team) -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation

1 0

openstack upgrades Thursday, 2024-04-25
by Andrew Bogott 25 Apr '24

25 Apr '24

I will be upgrading the cloud-vps openstack install on Thursday, beginning around 16:00 UTC. Here's what to expect: - Intermittent Horizon and API downtime (maybe an hour or two total) - Inability to schedule new VMs (also for an hour or two) Toolforge users will be unaffected by this outage. Existing, running services and VMs on cloud-vps should also be unaffected. In case you want to follow along at home, this is tracked as https://phabricator.wikimedia.org/T356287 -Andrew + the WMCS team

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce