October 2024 - Cloud - lists.wikimedia.org

Prometheus
by Roy Smith 14 Nov '24

14 Nov '24

I want to start exploring https://wikitech.wikimedia.org/wiki/Prometheus. Is there a test instance that I can use so I don't start throwing garbage at the production server?

3 2

Dumps failed delivery October dumps (frwiki, wikidata), pausing dumps generation
by ahoelzl＠wikimedia.org 30 Oct '24

30 Oct '24

Greetings XML Dump users, TL;DR: We are pausing the XML Dumps effective from now to correct runtime errors that we suspect are causing bad data dumps. We are working on a fix. Longer: Over the past couple months, we have noticed a growing amount of runtime errors coming from the process that generate the XML content Dumps. It has always been the case that this process may have transient issues and may miss some revisions some months, but the current situation has become such a recurring problem that we now suspect data corruption in recent dumps. We typically start a full dump (i.e. all revisions for all pages) on the 1st of the month for all wikis, and then we start a partial dump (i.e all current revisions for all pages) on the 20th of the month. Most of the October 1 2024 full runs are complete, except the French wiki and Wikidata wiki, which have failed for this month. The last successful copies of the French and Wikidata wikis are from September. All of the partial runs for the 20th of November are complete as well. However, any of these recent dumps may have underlying data quality issues. In the interest of not dumping potentially bad data, we have decided to pause the XML Dumps, effective for all future dumps from the date of this communication, until we find and fix the root cause of these errors. We acknowledge that many folks and downstream processes will be impacted and apologize for any inconvenience that this may cause you. We are prioritizing this work, and if interested, you can follow updates at https://phabricator.wikimedia.org/T377594. Feel free to open additional tickets if your use cases are affected, and do please link them to the main ticket. Further, if you have the ability, we welcome data quality analysis of recent dumps that you may have noticed in your use cases.

3 2

[Cloud-announce] Temporary accounts are coming, check if your tool may be affected
by Francesco Negri 30 Oct '24

30 Oct '24

If your tool does not read user information from Wiki Replicas, feel free to ignore this email. Temporary accounts [0] are starting to be rolled out, and since yesterday they are enabled on a few smaller wikis: Czech Wikiversity, Igbo Wikipedia, Italian Wikiquote, Swahili Wikipedia, and Serbo-Croatian Wikipedia. [1] Temporary Accounts modify the ways MediaWiki stores anonymous users in database tables. If you manage a tool that reads user information for anonymous users, you should check the page "How should I update my code?" [2] to find out if you need to make changes to your tool. You can use the wikis listed above to test that your tool is working correctly. If you have questions or if you want to report an issue, you can file a Phabricator task as a subtask of [3]. [0] https://www.mediawiki.org/wiki/Trust_and_Safety_Product/Temporary_Accounts [1] https://www.mediawiki.org/wiki/Trust_and_Safety_Product/Temporary_Accounts/… [2] https://www.mediawiki.org/wiki/Trust_and_Safety_Product/Temporary_Accounts/… [3] https://phabricator.wikimedia.org/T378516 -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

Puppet failures?
by Travis Briggs 23 Oct '24

23 Oct '24

Hi, I woke up this morning to "Puppet failure on...." alerts for every cloud VPS server I have access to. I basically just ignored it because I assumed that many failures was something systemic, and I would wait for an announcement from the cloud services team. Before I try to debug something this gnarly, *is* there something going on? Or is it just coincidence that 8 machines across two projects got these failures this morning? Thanks, -Travis

2 2

Fwd: [Cloud VPS alert][wikispeech] Puppet failure on producer.wikispeech.eqiad1.wikimedia.cloud (172.16.0.200)
by Sebastian Berlin 22 Oct '24

22 Oct '24

I received this email for four instances in two different projects. Running `sudo run-puppet-agent` gave the same error. Do I need to do anything? *Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE) E-post/*E-Mail*: sebastian.berlin(a)wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84 ---------- Forwarded message --------- From: root <root(a)producer.wikispeech.eqiad1.wikimedia.cloud> Date: Tue, 22 Oct 2024 at 10:16 Subject: [Cloud VPS alert][wikispeech] Puppet failure on producer.wikispeech.eqiad1.wikimedia.cloud (172.16.0.200) To: <sebastian.berlin(a)wikimedia.se> Puppet is having issues on the "producer.wikispeech.eqiad1.wikimedia.cloud (172.16.0.200)" instance in project wikispeech in Wikimedia Cloud VPS. Puppet is running with failures. Working Puppet runs are needed to maintain instance security and logins. As long as Puppet continues to fail, this system is in danger of becoming unreachable. You are receiving this email because you are listed as member for the project that contains this instance. Please take steps to repair this instance or contact a Cloud VPS admin for assistance. If your host is expected to fail puppet runs and you want to disable this alert, you can create a file under /.no-puppet-checks, that will skip the checks. You might find some help here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Cloud_V… For further support, visit #wikimedia-cloud on libera.chat or <https://wikitech.wikimedia.org> Some extra info follows: ---- Last run summary: application: converged_environment: production initial_environment: production run_mode: agent changes: total: 0 events: failure: 0 success: 0 total: 0 resources: changed: 0 corrective_change: 0 failed: 0 failed_to_restart: 0 out_of_sync: 0 restarted: 0 scheduled: 0 skipped: 0 total: 0 time: fact_generation: 0.5404427628964186 last_run: 1729583430 plugin_sync: 0.7785281259566545 startup_time: 0.652441437 total: 2.206511162 version: config: null puppet: 7.23.0 ---- Failed resources if any: No failed resources. --- Last run log: ERR: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node producer.wikispeech.eqiad1.wikimedia.cloud: Exception while executing '/usr/local/bin/puppet-enc': Cannot run program "/usr/local/bin/puppet-enc" (in directory "."): error=0, Failed to exec spawn helper: pid: 869822, exit value: 1 WARNING: Not using cache on failed catalog ERR: Could not retrieve catalog; skipping run ---- Exceptions that happened when running the script if any: No exceptions happened.

4 3

New Code of Conduct Committee members
by Amir Sarabadani 14 Oct '24

14 Oct '24

Hello everyone, The Committee has finished selecting its new members. The new committee candidates are (in alphabetical order): - Amir Sarabadani - Egbe Eugene Agbor (Eugene233) - Greg Grossmeier - Jayprakash12345 - Kamila Součková Auxiliary members will be (alphabetically): - Effie Mouzeli - Joris Darlington Quarshie - Martin Urbanec - MusikAnimal - SD0001 You can read more information about the members at [1]. List of changes from the last term: - Greg and Kamila will be joining the main committee. - Nuria is leaving the main committee - MusikAnimal is moving to the aux committee - Tony is leaving the aux committee This is not the final structure. According to the CoC [2], the current committee publishes the new members and calls for public feedback for *six weeks* and after that, the current committee might apply changes to the structure based on public feedback. Please let the committee know (via techconduct(a)wikimedia.org) if you have any concern regarding the members and its structure until *08 October 2024* and after that, the new committee will be in effect and will serve for a year. [1] https://www.mediawiki.org/wiki/Code_of_Conduct/Committee/Members/Candidates [2] https://www.mediawiki.org/wiki/Code_of_Conduct/Committee#Selection_of_new_m… -- Amir (he/him)

1 1

What do y'all use Dumps for?
by Bryan Davis 10 Oct '24

10 Oct '24

I was asked recently what I knew about the types of tools that use data from the https://dumps.wikimedia.org/ project. I had to admit that I really didn't know of many tools off the top of my head that relied on dumps. Most of the use cases I have heard about are for research topics like looking at word frequencies and sentence complexity, or machine learning things that consume some or all of the wiki corpus. Do you run a tool that needs data from Dumps to do its job? I would love to hear some stories about how this data helps folks advance the work of the movement. Bryan -- Bryan Davis Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

10 9

Fwd: [Wikitech-l] Forthcoming maintenance to irc.wikimedia.org - Moving to ircstream
by Bryan Davis 09 Oct '24

09 Oct '24

Forwarding from wikitech-l for visibility. TL;DR - irc.wikimedia.org is getting a new backend software stack on 2024-10-10. No client facing changes are expected. See https://phabricator.wikimedia.org/T376014 for technical details and to report issues. ---------- Forwarded message --------- From: Moritz Mühlenhoff <mmuhlenhoff(a)wikimedia.org> Date: Tue, Oct 8, 2024 at 10:27 AM Subject: [Wikitech-l] Forthcoming maintenance to irc.wikimedia.org - Moving to ircstream To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org> Hi all, we're moving irc.wikimedia.org to a new infrastructure [1]. This service is unrelated to the IRC network (Libera) we use for real time discussions. Instead irc.wikimedia.org is an IRC service for broadcasting recent changes events from public Wikimedia wikis to be used by various bots connected to per-wiki IRC channels. irc.wikimedia.org should not be used for any new bots (which should rather use Eventstreams[2]), but we still have various important bots relying on the legacy IRC-based infrastructure. The current setup is full of technical debt and ultimately based on a patched version of a very old release of ircd-ratbox with a relay service written in Python 2. We are replacing it with a modern standalone implementation in Python, which broadcasts IRC notifications in a format compatible to what is currently in use by the legacy setup: https://github.com/paravoid/ircstream Last week during the SRE Infrastructure Foundations hackathon the ircstream production setup was created. All our tests have been successful, so on Thursday October 10 at 08:00 UTC we'll switch the irc.wikimedia.org DNS name to the new setup. No changes are needed to any bots, but if you run into any issues after the switch, please notify us in the #wikimedia-sre-foundations IRC channel or leave a note at https://phabricator.wikimedia.org/T376014. Cheers, Luca, Simon and Moritz Footnotes: [1] https://wikitech.wikimedia.org/wiki/Irc.wikimedia.org [2] https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service _______________________________________________ Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ -- Bryan Davis Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

[Cloud-announce] [Toolforge maintenance 2024-10-08] Tomorrow at 12:00 UTC toolforge upgrade
by David Caro 08 Oct '24

08 Oct '24

Hi! We will be upgrading toolforge build service tomorrow October 8th at 12:00 UTC. There's no downtime expected, but the process this time is a bit flaky, and if something goes wrong, it might make any new builds to fail, or prevent you from accessing the logs of old ones until manually fixed. I'll notify replying to this email and on irc before and after the upgrade. Thanks! -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 3

Introducing Tool-Watch: Your Feedback Needed!
by Vasanth Gopa 04 Oct '24

04 Oct '24

Dear Tool Maintainers of Toolforge, The idea for Tool-Watch <https://tool-watch.toolforge.org/> originated during the Wikimania Hackathon 2023. A group of us was exploring various tools from Toolhub and noticed that many of the links we clicked were not functioning. This sparked our interest in finding an existing monitoring tool for tools at Toolforge, but we were unable to locate one. Motivated by this experience, we began developing Tool-Watch, which has been live for some time now. As part of this initiative, we are considering implementing email notifications for tool maintainers. The goal is to notify maintainers if their tool has been down for an extended period, allowing for timely intervention and maintenance. We believe this will help keep our tools running smoothly and reduce unnecessary infrastructure costs and make the catalogue cleaner. We value your input and would love to hear your thoughts on this topic. Specifically, we’d like to know: - Would you find email notifications helpful if they are triggered only after extended downtime? - Should the notification be triggered after 3, 7, or 30 days of downtime? Or do you have any other time frame in mind? Please feel free to share your feedback by replying to this email or joining the discussion on Phabricator T368816#10201603 <https://phabricator.wikimedia.org/T368816#10201603>. Also, special thanks to Soham and other contributors for supporting me in building this tool! Tool: https://tool-watch.toolforge.org/ Phabricator discussion: https://phabricator.wikimedia.org/T368816 Thank you for your continued support and dedication to our community. We look forward to hearing from you! Regards, Gopa Vasanth <https://gopavasanth.github.io/> Twitter <https://twitter.com/gopavasanth1999> | LinkedIn <https://www.linkedin.com/in/gopa-vasanth/> | GitHub <https://github.com/gopavasanth> | Gerrit <https://gerrit.wikimedia.org/r/#/q/gopavasanth> “Yesterday is not ours to recover, but tomorrow is ours to win or lose.”

1 0

2024

2023

2022

2021

2020

2019

2018

2017

Cloud October 2024