- Cloud - lists.wikimedia.org

Puppet failures?
by Travis Briggs 23 Oct '24

23 Oct '24

Hi, I woke up this morning to "Puppet failure on...." alerts for every cloud VPS server I have access to. I basically just ignored it because I assumed that many failures was something systemic, and I would wait for an announcement from the cloud services team. Before I try to debug something this gnarly, *is* there something going on? Or is it just coincidence that 8 machines across two projects got these failures this morning? Thanks, -Travis

2 2

Fwd: [Cloud VPS alert][wikispeech] Puppet failure on producer.wikispeech.eqiad1.wikimedia.cloud (172.16.0.200)
by Sebastian Berlin 22 Oct '24

22 Oct '24

I received this email for four instances in two different projects. Running `sudo run-puppet-agent` gave the same error. Do I need to do anything? *Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE) E-post/*E-Mail*: sebastian.berlin(a)wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84 ---------- Forwarded message --------- From: root <root(a)producer.wikispeech.eqiad1.wikimedia.cloud> Date: Tue, 22 Oct 2024 at 10:16 Subject: [Cloud VPS alert][wikispeech] Puppet failure on producer.wikispeech.eqiad1.wikimedia.cloud (172.16.0.200) To: <sebastian.berlin(a)wikimedia.se> Puppet is having issues on the "producer.wikispeech.eqiad1.wikimedia.cloud (172.16.0.200)" instance in project wikispeech in Wikimedia Cloud VPS. Puppet is running with failures. Working Puppet runs are needed to maintain instance security and logins. As long as Puppet continues to fail, this system is in danger of becoming unreachable. You are receiving this email because you are listed as member for the project that contains this instance. Please take steps to repair this instance or contact a Cloud VPS admin for assistance. If your host is expected to fail puppet runs and you want to disable this alert, you can create a file under /.no-puppet-checks, that will skip the checks. You might find some help here: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Cloud_V… For further support, visit #wikimedia-cloud on libera.chat or <https://wikitech.wikimedia.org> Some extra info follows: ---- Last run summary: application: converged_environment: production initial_environment: production run_mode: agent changes: total: 0 events: failure: 0 success: 0 total: 0 resources: changed: 0 corrective_change: 0 failed: 0 failed_to_restart: 0 out_of_sync: 0 restarted: 0 scheduled: 0 skipped: 0 total: 0 time: fact_generation: 0.5404427628964186 last_run: 1729583430 plugin_sync: 0.7785281259566545 startup_time: 0.652441437 total: 2.206511162 version: config: null puppet: 7.23.0 ---- Failed resources if any: No failed resources. --- Last run log: ERR: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node producer.wikispeech.eqiad1.wikimedia.cloud: Exception while executing '/usr/local/bin/puppet-enc': Cannot run program "/usr/local/bin/puppet-enc" (in directory "."): error=0, Failed to exec spawn helper: pid: 869822, exit value: 1 WARNING: Not using cache on failed catalog ERR: Could not retrieve catalog; skipping run ---- Exceptions that happened when running the script if any: No exceptions happened.

4 3

New Code of Conduct Committee members
by Amir Sarabadani 14 Oct '24

14 Oct '24

Hello everyone, The Committee has finished selecting its new members. The new committee candidates are (in alphabetical order): - Amir Sarabadani - Egbe Eugene Agbor (Eugene233) - Greg Grossmeier - Jayprakash12345 - Kamila Součková Auxiliary members will be (alphabetically): - Effie Mouzeli - Joris Darlington Quarshie - Martin Urbanec - MusikAnimal - SD0001 You can read more information about the members at [1]. List of changes from the last term: - Greg and Kamila will be joining the main committee. - Nuria is leaving the main committee - MusikAnimal is moving to the aux committee - Tony is leaving the aux committee This is not the final structure. According to the CoC [2], the current committee publishes the new members and calls for public feedback for *six weeks* and after that, the current committee might apply changes to the structure based on public feedback. Please let the committee know (via techconduct(a)wikimedia.org) if you have any concern regarding the members and its structure until *08 October 2024* and after that, the new committee will be in effect and will serve for a year. [1] https://www.mediawiki.org/wiki/Code_of_Conduct/Committee/Members/Candidates [2] https://www.mediawiki.org/wiki/Code_of_Conduct/Committee#Selection_of_new_m… -- Amir (he/him)

1 1

What do y'all use Dumps for?
by Bryan Davis 10 Oct '24

10 Oct '24

I was asked recently what I knew about the types of tools that use data from the https://dumps.wikimedia.org/ project. I had to admit that I really didn't know of many tools off the top of my head that relied on dumps. Most of the use cases I have heard about are for research topics like looking at word frequencies and sentence complexity, or machine learning things that consume some or all of the wiki corpus. Do you run a tool that needs data from Dumps to do its job? I would love to hear some stories about how this data helps folks advance the work of the movement. Bryan -- Bryan Davis Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

10 9

Fwd: [Wikitech-l] Forthcoming maintenance to irc.wikimedia.org - Moving to ircstream
by Bryan Davis 08 Oct '24

08 Oct '24

Forwarding from wikitech-l for visibility. TL;DR - irc.wikimedia.org is getting a new backend software stack on 2024-10-10. No client facing changes are expected. See https://phabricator.wikimedia.org/T376014 for technical details and to report issues. ---------- Forwarded message --------- From: Moritz Mühlenhoff <mmuhlenhoff(a)wikimedia.org> Date: Tue, Oct 8, 2024 at 10:27 AM Subject: [Wikitech-l] Forthcoming maintenance to irc.wikimedia.org - Moving to ircstream To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org> Hi all, we're moving irc.wikimedia.org to a new infrastructure [1]. This service is unrelated to the IRC network (Libera) we use for real time discussions. Instead irc.wikimedia.org is an IRC service for broadcasting recent changes events from public Wikimedia wikis to be used by various bots connected to per-wiki IRC channels. irc.wikimedia.org should not be used for any new bots (which should rather use Eventstreams[2]), but we still have various important bots relying on the legacy IRC-based infrastructure. The current setup is full of technical debt and ultimately based on a patched version of a very old release of ircd-ratbox with a relay service written in Python 2. We are replacing it with a modern standalone implementation in Python, which broadcasts IRC notifications in a format compatible to what is currently in use by the legacy setup: https://github.com/paravoid/ircstream Last week during the SRE Infrastructure Foundations hackathon the ircstream production setup was created. All our tests have been successful, so on Thursday October 10 at 08:00 UTC we'll switch the irc.wikimedia.org DNS name to the new setup. No changes are needed to any bots, but if you run into any issues after the switch, please notify us in the #wikimedia-sre-foundations IRC channel or leave a note at https://phabricator.wikimedia.org/T376014. Cheers, Luca, Simon and Moritz Footnotes: [1] https://wikitech.wikimedia.org/wiki/Irc.wikimedia.org [2] https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service _______________________________________________ Wikitech-l mailing list -- wikitech-l(a)lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave(a)lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/ -- Bryan Davis Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

[Cloud-announce] [Toolforge maintenance 2024-10-08] Tomorrow at 12:00 UTC toolforge upgrade
by David Caro 08 Oct '24

08 Oct '24

Hi! We will be upgrading toolforge build service tomorrow October 8th at 12:00 UTC. There's no downtime expected, but the process this time is a bit flaky, and if something goes wrong, it might make any new builds to fail, or prevent you from accessing the logs of old ones until manually fixed. I'll notify replying to this email and on irc before and after the upgrade. Thanks! -- David Caro SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 3

Introducing Tool-Watch: Your Feedback Needed!
by Vasanth Gopa 04 Oct '24

04 Oct '24

Dear Tool Maintainers of Toolforge, The idea for Tool-Watch <https://tool-watch.toolforge.org/> originated during the Wikimania Hackathon 2023. A group of us was exploring various tools from Toolhub and noticed that many of the links we clicked were not functioning. This sparked our interest in finding an existing monitoring tool for tools at Toolforge, but we were unable to locate one. Motivated by this experience, we began developing Tool-Watch, which has been live for some time now. As part of this initiative, we are considering implementing email notifications for tool maintainers. The goal is to notify maintainers if their tool has been down for an extended period, allowing for timely intervention and maintenance. We believe this will help keep our tools running smoothly and reduce unnecessary infrastructure costs and make the catalogue cleaner. We value your input and would love to hear your thoughts on this topic. Specifically, we’d like to know: - Would you find email notifications helpful if they are triggered only after extended downtime? - Should the notification be triggered after 3, 7, or 30 days of downtime? Or do you have any other time frame in mind? Please feel free to share your feedback by replying to this email or joining the discussion on Phabricator T368816#10201603 <https://phabricator.wikimedia.org/T368816#10201603>. Also, special thanks to Soham and other contributors for supporting me in building this tool! Tool: https://tool-watch.toolforge.org/ Phabricator discussion: https://phabricator.wikimedia.org/T368816 Thank you for your continued support and dedication to our community. We look forward to hearing from you! Regards, Gopa Vasanth <https://gopavasanth.github.io/> Twitter <https://twitter.com/gopavasanth1999> | LinkedIn <https://www.linkedin.com/in/gopa-vasanth/> | GitHub <https://github.com/gopavasanth> | Gerrit <https://gerrit.wikimedia.org/r/#/q/gopavasanth> “Yesterday is not ours to recover, but tomorrow is ours to win or lose.”

1 0

Request for my Cloud VPS project review
by Sulav K Shetri 30 Sep '24

30 Sep '24

I had requested an new Cloud VPS project 6 days back and waiting for review or approval but it has been 6 days since my request but till now no one has reviewed it so I wanted to know when will be reviewed and this is the link of my request Request creation of createwikitest VPS project <https://phabricator.wikimedia.org/T375454> So I would be happy to my request be reviewed as soon as possible.

2 2

[Cloud-announce] Toolforge kubernetes upgrade to 1.27
by David Caro 16 Sep '24

16 Sep '24

Hello everyone!, As part of the Toolforge upgarde workgroup [0], we will be upgrading Toolforge Kubernetes to v1.27 today, Monday Sep 16 around 1:00PM UTC. We do not expect any downtime, but some jobs and webservices may restart as they get shuffled around to different worker nodes. Please report any issues you encounter.[1] For details see: https://phabricator.wikimedia.org/T359641 [0] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Ongoing_Efforts/Toolfo… [1] https://wikitech.wikimedia.org/wiki/Portal:Toolforge#Communication_and_supp… Cheers, -- David Caro on behalf of Ndibe Raymond Olisaemeka SRE - Cloud Services Wikimedia Foundation <https://wikimediafoundation.org/> PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3 "Imagine a world in which every single human being can freely share in the sum of all knowledge. That's our commitment." _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

[Cloud-announce] Quarry to remove results more than 90 days old
by Vivian Rook 14 Sep '24

14 Sep '24

Hello, quarry currently retains query results forever. This proves to be a problem when removed data is retained in quarry. In order to improve this while still leaving results accessible, results older than 90 days will be removed. The queries will not be impacted by this change. And the results can of course be regenerated (with current data) at any time by resubmitting the query. https://phabricator.wikimedia.org/T360041 The plan is for this to go into effect in later October (2024-10-21), could go in earlier but I won't be here until then. Thank you! -- *Vivian Rook (They/Them)* Site Reliability Engineer Wikimedia Foundation <https://wikimediafoundation.org/> _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud