Cloud-announce November 2024

cloud-announce@lists.wikimedia.org

5 participants
6 discussions

[wiki replicas] Expected replication lag on s8 (wikidatawiki)
by Francesco Negri 03 Dec '24

03 Dec '24

If you are using wiki replicas to query the "wikidatawiki" database (section s8), please be aware that we are expecting replication lag to grow up to 10 days, because of some ongoing maintenance work [0]. Only section s8 is affected, which contains only the "wikidatawiki" database. [1] Queries against other databases should not see any lag. This is going to impact tools using wiki replicas, as well as queries running on Quarry or PAWS. You can check the current replication lag at https://replag.toolforge.org/ Thanks for your patience while we complete this maintenance work. [0] https://phabricator.wikimedia.org/T367856 [1] https://noc.wikimedia.org/db.php#tabs-s8 -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

2 1

[Incident] Toolforge: Ongoing intermittent DNS resolution issues
by Slavina Stefanova 26 Nov '24

26 Nov '24

Hello, We are currently investigating widespread intermittent DNS resolution issues within the Toolforge Kubernetes cluster that began on Sunday. These issues are causing some jobs and deployments to fail, particularly on NFS worker nodes. Impact: - Some tools may experience failed deployments or crashes - Job execution may be inconsistent - Image pulls may fail intermittently Our team is actively investigating and working to resolve the issue. We will send an update once we have more information or when the incident is resolved. Thank you for your patience, Slavina, on behalf of the WMCS Team -- Slavina Stefanova (she/her) Software Engineer | Cloud Services Wikimedia Foundation

1 1

ToolsDB is getting upgraded to MariaDB 10.6
by Francesco Negri 25 Nov '24

25 Nov '24

Next Monday, November 25th, at around 13:00 UTC, ToolsDB will be upgraded from MariaDB v10.4.29 to MariaDB v10.6.19. [0] I have already created a new host "tools-db-4" that is running the new version, and is replicating from the current primary. On Monday, I will fail over the current primary to the new host. [1] All connections will be dropped and the DNS will be updated to point to the new host. Tools should automatically reconnect to the new host. No downtime is expected but there will be a few minutes of read-only time. For most tools, the upgrade should be painless. However, you might want to check the official docs listing incompatible changes introduced in version 10.5 [2] and 10.6. [3] If you find any issues, please let us know in the #wikimedia-cloud IRC channel. [0] https://phabricator.wikimedia.org/T352206 [1] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/ToolsDB#Failing_… [2] https://mariadb.com/kb/en/upgrading-from-mariadb-10-4-to-mariadb-10-5/ [3] https://mariadb.com/kb/en/upgrading-from-mariadb-10-5-to-mariadb-10-6/ -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation

1 2

Buster VMs to be deleted on Friday
by Andrew Bogott 13 Nov '24

13 Nov '24

On Friday I will delete the following VMs. All are running the long-deprecated Debian Buster OS and have been shut down for several months without user response or complaint. Please respond directly to me if you need any of the above to be preserved in some form. Context can be found at https://phabricator.wikimedia.org/T331738. centralnotice-staging: cn-staging-3.centralnotice-staging.eqiad1.wikimedia.cloud commons-corruption-checker: main.commons-corruption-checker.eqiad1.wikimedia.cloud deployment-prep: deployment-docker-proton01.deployment-prep.eqiad1.wikimedia.cloud deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud deployment-maps-master01.deployment-prep.eqiad1.wikimedia.cloud deployment-poolcounter06.deployment-prep.eqiad1.wikimedia.cloud deployment-restbase04.deployment-prep.eqiad1.wikimedia.cloud etytree: etytree-a.etytree.eqiad1.wikimedia.cloud mediawiki-vagrant: mwv-builder-03.mediawiki-vagrant.eqiad1.wikimedia.cloud schematreerecommender: recommender.schematreerecommender.eqiad1.wikimedia.cloud stress-tester.schematreerecommender.eqiad1.wikimedia.cloud wikicommunityhealth: backend.wikicommunityhealth.eqiad1.wikimedia.cloud frontend.wikicommunityhealth.eqiad1.wikimedia.cloud wikispore: wikispore-prod.wikispore.eqiad1.wikimedia.cloud

1 0

New Toolforge Standards Committee members have been seated
by Bryan Davis 05 Nov '24

05 Nov '24

Following a public nomination process[0], vetting by various groups[1], and signing of non-disclosure agreements with the Wikimedia Foundation[2] the following folks have been seated as the newest incarnation of the Toolforge Standards Committee[3]: * JJMC89 * Pintoch * Lucas Werkmeister * Legoktm * Waldyrious * SD0001 * MusikAnimal * Theprotonade These folks are now empowered to help resolve requests related to the Right to Fork policy[4] and the Abandoned tool policy[5]. They may also choose as a group to promote other standards and practices within Toolforge that are intended to help us all make and support better tools for the community. [0]: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.… [1]: https://phabricator.wikimedia.org/T370474 [2]: https://phabricator.wikimedia.org/T374993 [3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Toolforge_standards_comm… [4]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Right_to_fork_policy [5]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy Bryan -- Bryan Davis Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Toolforge Kubernetes upgrade to 1.28
by Raymond Ndibe 04 Nov '24

04 Nov '24

Hello everyone!, We will be upgrading Toolforge Kubernetes to v1.28 on Monday Oct 28 around 1:00PM UTC. We do not expect any downtime, but some jobs and webservices may restart as they get shuffled around to different worker nodes. Please report any issues you encounter.[0] For details see: https://phabricator.wikimedia.org/T362867 [0] https://wikitech.wikimedia.org/wiki/Portal:Toolforge#Communication_and_supp… Cheers, -- Ndibe Raymond Olisaemeka Software Engineer II - Technical Engagement Wikimedia Foundation <https://wikimediafoundation.org/>

1 3

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-announce November 2024