Cloud December 2025

cloud@lists.wikimedia.org

7 participants
6 discussions

[Cloud-announce] New Toolforge Standards Committee members appointed
by JJMC89 05 Jan '26

05 Jan '26

Following a public nomination process[0], request for feedback[1], and signing of non-disclosure agreements with the Wikimedia Foundation[2] the following users have been appointed as the newest incarnation of the Toolforge Standards Committee[3]: * JJMC89 * Lucas Werkmeister * Pintoch * SD0001 * Sohom Datta They are now empowered to help resolve requests related to the Right to fork policy[4] and the Abandoned tool policy[5]. They may also choose as a group to promote other standards and practices within Toolforge that are intended to help us all make and support better tools for the community. [0]: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.… [1]: https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/messa… [2]: https://phabricator.wikimedia.org/T413434 [3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Toolforge_standards_comm… [4]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Right_to_fork_policy [5]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 1

502 Bad Gateway error in public-paws.wmcloud.org
by delahera＠gmail.com 23 Dec '25

23 Dec '25

Hi all! I'm working with some notebooks on PAWS and sometimes I get a 502 Bad Gateway error when trying to access their HTML version via public-paws.wmcloud.org. See for example https://public-paws.wmcloud.org/User:Diegodlh/wmuy-caad/semantic.ipynb. You should still be able to get the raw version appending `?format=raw` at the end. I think it may have something to do with the notebook's output or metadata, since the problem seems to go away when clearing the outputs (but usually reappears after running the notebook again). I thought I had identified some cells causing the issue, so I added some code to clear the output after running (look for `clear_output()` in the raw notebook above). This seemed to work some time, but it's failing again now. My understanding about how PAWS works internally is very basic and I don't know what else I can do to debug this. I searched for this problem in Phabricator, this mailing list and the Telegram channel, but didn't find anything. Could someone help me see what may be going on and how to fix it? Thanks!

1 0

Resource consumption on toolforge
by Nicolas Vervelle 07 Dec '25

07 Dec '25

Hi everyone, I'm seeing strange things with resource consumption with toolforge. My resource consumption reported by the quota is way above the consumption of my currently running job, preventing other jobs from starting. Where does the extra consumption come from ? And how to clean this ? More detailed data below Any help appreciated, NicoV My tool account is wpcleaner on toolforge. Only one job is currently running : tools.wpcleaner@tools-bastion-15:~$ toolforge jobs list +------------------------+--------------------+------------------------------------------+ | Job name: | Job type: | Status: | +------------------------+--------------------+------------------------------------------+ | wpcleaner-cs-weekly | schedule: @weekly | Last schedule time: 2025-12-03T05:17:00Z | | wpcleaner-en-list | schedule: @weekly | Unable to start, out of quota for memory | | wpcleaner-fr-dab | schedule: @weekly | Unable to start, out of quota for memory | | wpcleaner-fr-daily | schedule: @daily | Last schedule time: 2025-12-07T03:58:00Z | | wpcleaner-fr-list | schedule: @weekly | Running for 2d4h10m | | wpcleaner-fr-weekly | schedule: @weekly | Last schedule time: 2025-12-04T05:03:00Z | | wpcleaner-meta-list | schedule: @monthly | Last schedule time: 2025-12-04T04:25:00Z | | wpcleaner-meta-monthly | schedule: @monthly | Last schedule time: 2025-11-18T06:05:00Z | +------------------------+--------------------+------------------------------------------+ The currently running job defines 3G and 1 CPU for resources : tools.wpcleaner@tools-bastion-15:~$ toolforge jobs show wpcleaner-fr-list +---------------+-----------------------------------------------------------------+ | Job name: | wpcleaner-fr-list | +---------------+-----------------------------------------------------------------+ | Command: | /data/project/wpcleaner/tools/scripts/fr_ListCheckWiki_List.sh | +---------------+-----------------------------------------------------------------+ | Job type: | schedule: @weekly | +---------------+-----------------------------------------------------------------+ | Image: | jdk17 | +---------------+-----------------------------------------------------------------+ | Port: | none | +---------------+-----------------------------------------------------------------+ | File log: | no | +---------------+-----------------------------------------------------------------+ | Output log: | | +---------------+-----------------------------------------------------------------+ | Error log: | | +---------------+-----------------------------------------------------------------+ | Emails: | onfailure | +---------------+-----------------------------------------------------------------+ | Resources: | mem: 3.0Gi, cpu: 1.0 | +---------------+-----------------------------------------------------------------+ | Replicas: | | +---------------+-----------------------------------------------------------------+ | Mounts: | none | +---------------+-----------------------------------------------------------------+ | Retry: | no | +---------------+-----------------------------------------------------------------+ | Timeout: | no | +---------------+-----------------------------------------------------------------+ | Health check: | none | +---------------+-----------------------------------------------------------------+ | Status: | Running for 2d4h11m | +---------------+-----------------------------------------------------------------+ | Hints: | Last run at 2025-12-03T20:36:14Z. Pod in 'Running' phase. State | | | 'running'. Started at '2025-12-03T20:36:15Z'. | +---------------+-----------------------------------------------------------------+ But the quota command says I'm consuming 2.5 CPU and 6.5G, which is a lot more than what the only running job defines. tools.wpcleaner@tools-bastion-15:~$ toolforge jobs quota Running jobs Used Limit -------------------------------------------- ------ ------- Total running jobs at once (Kubernetes pods) 3 16 Running one-off and cron jobs 4 15 CPU 2.5 16.0 Memory 6.5Gi 8.0Gi Per-job limits Used Limit ---------------- ------ ------- CPU 3.0 Memory 6.0Gi Job definitions Used Limit ---------------------------------------- ------ ------- Cron jobs 8 50 Continuous jobs (including web services) 1 16

2 3

[Cloud-announce] Toolforge standards committee nominations open
by JJ Clark 02 Dec '25

02 Dec '25

TLDR: Nominations are open for the Toolforge standards committee[0] at https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Toolforge_standards… Following the TFSC's revision to the appointment process[1] from appointment as needed by the Foundation to an annual appointment by the existing committee, this is the first annual appointment nomination period. Nominations are being collected on Wikitech[2] through 2025-11-28. The committee's primary duties are to oversee the Right to fork policy[3] and Abandoned tool policy[4]. Any Toolforge project member in good standing is eligible for nomination. Self-nominations are acceptable. Nominations by a third party must be counter-signed by the nominee to indicate they would accept the responsibility and be willing to sign the Volunteer NDA[5]. Following the nomination period there will be a two-week period for the Toolforge admins and community to provide feedback on the nominees to the current committee. Following the feedback period, the current committee will vote on the nominees. Nominees approved by the committee must sign the Volunteer NDA to be appointed to the committee. [0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Toolforge_standards_comm… [1]: https://wikitech.wikimedia.org/wiki/Special:Diff/2357725 [2]: https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Toolforge_standards… [3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Right_to_fork_policy [4]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy [5]: https://wikitech.wikimedia.org/wiki/Volunteer_NDA _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 1

GitLab/Toolforge deployment error
by jjpmaster 01 Dec '25

01 Dec '25

Hi! I'm deploying a tool to Toolforge using the new GitLab CI system, and I got the following error: gotten_deployment='{"messages":{"info":[],"warning":["You are using a beta feature of Toolforge."],"error":["Unable to find namespace tool-jjpmaster-bot-enwb-t2 or config jjpmaster-bot-enwb-t2-config for jjpmaster-bot-enwb-t2"]}}'The name of the tool is "jjpmaster-bot-enwb-t2" (see https://gitlab.wikimedia.org/toolforge-repos/jjpmaster-bot-enwb-t2/-/jobs/6…). Does anyone know what this error could represent? Thank you, JJPMaster

2 2

[Cloud-announce] Reboot (some of) your instances, or expect them to be rebooted for you on December 2
by Andrew Bogott 01 Dec '25

01 Dec '25

tl;dr: All cloud-vps instances listed in this email will be rebooted, by me, on Tuesday December 2nd. This list includes several bastion hosts, where existing logins, long-running jobs, or tmux sessions will be interrupted. If you would like more control, you can hard reboot your instances anytime in the next week and then I will not need to reboot them on the 2nd. Make sure to use the 'Hard reboot instance' menu option; a soft reboot is not enough to clear the network settings. details: We have recently updated our virtualization hardware to slightly increase the MTU[0], allowing normal-sized packet transmission[1]. This change was necessary in order to resolve an issue with Docker containers and, potentially, other issues not yet revealed. Certain VMs that existed before this update now have an MTU size that doesn't match the new size set on all hypervisors; this limits our ability to migrate those VMs and perform other routine maintenance tasks. A stop/start or 'hard reboot' of affected instances resets the MTU, as it rebuilds the virtual network stack associated with the host. A 'soft' reboot (or simply rebooting from within the OS of the instance) leaves the network stack intact so does not resolve the issue. If you want to reboot your VMs before Tuesday, there's no need to notify WMCS staff. I'll re-run the report that lists affected VMs immediately before rebooting, and a hard reboot will remove your VMs from any future such report. -Andrew [0] https://en.wikipedia.org/wiki/Maximum_transmission_unit [1] https://phabricator.wikimedia.org/T408543 ===== accounts-appserver7.account-creation-assistance.eqiad1.wikimedia.cloud deep-dive.analytics.eqiad1.wikimedia.cloud gitlab-docker-runner-v2.analytics.eqiad1.wikimedia.cloud T389375.appservers.eqiad1.wikimedia.cloud rn-hcptchprxy-pki-01.appservers.eqiad1.wikimedia.cloud rn-hcptchprxy-puppet-01.appservers.eqiad1.wikimedia.cloud rn-hcptchprxy-puppetdb-01.appservers.eqiad1.wikimedia.cloud rn-hcptchprxy-urldownloader-[01-02].appservers.eqiad1.wikimedia.cloud bastion-eqiad1-[5-6].bastion.eqiad1.wikimedia.cloud k3s.catalyst.eqiad1.wikimedia.cloud k3s-envDB.catalyst.eqiad1.wikimedia.cloud k3s-worker[01-02].catalyst.eqiad1.wikimedia.cloud k3s-worker01.catalyst-dev.eqiad1.wikimedia.cloud ntp-[5-6].cloudinfra.eqiad1.wikimedia.cloud mediawiki2latex.collection-alt-renderer.eqiad1.wikimedia.cloud copypatrol-backend-prod-02.copypatrol.eqiad1.wikimedia.cloud cvn-apache11.cvn.eqiad1.wikimedia.cloud cvn-app[13-14].cvn.eqiad1.wikimedia.cloud deployment-poolcounter07.deployment-prep.eqiad1.wikimedia.cloud gitlab-1002.devtools.eqiad1.wikimedia.cloud gitlab-runner-[1007-1008].devtools.eqiad1.wikimedia.cloud k3s-test.devtools.eqiad1.wikimedia.cloud generator01.dumpstorrents.eqiad1.wikimedia.cloud debian13-test.dwl.eqiad1.wikimedia.cloud taxonbot4.dwl.eqiad1.wikimedia.cloud tmp.entity-detection.eqiad1.wikimedia.cloud runner-[1031-1040].gitlab-runners.eqiad1.wikimedia.cloud wikiwho-dev.globaleducation.eqiad1.wikimedia.cloud prod0.hashtags.eqiad1.wikimedia.cloud language-lab.language.eqiad1.wikimedia.cloud lpl-cx-sx2.language.eqiad1.wikimedia.cloud lpl-recommend.language.eqiad1.wikimedia.cloud lpl-services.language.eqiad1.wikimedia.cloud logging-logstash-04.logging.eqiad1.wikimedia.cloud mariadbcompiler-trixie.mariadbtest.eqiad1.wikimedia.cloud ci2.mediawiki-quickstart.eqiad1.wikimedia.cloud coder-env-1.mobileappsperformance.eqiad1.wikimedia.cloud wikiapiary.mwstake.eqiad1.wikimedia.cloud filippo-centrallog-02.o11y.eqiad1.wikimedia.cloud filippo-cloudcephosd-01.o11y.eqiad1.wikimedia.cloud filippo-clouddumps-01.o11y.eqiad1.wikimedia.cloud filippo-cloudgw-01.o11y.eqiad1.wikimedia.cloud filippo-cloudvirt-[01-02].o11y.eqiad1.wikimedia.cloud phi-alert-01.o11y.eqiad1.wikimedia.cloud phi-arclamp-01.o11y.eqiad1.wikimedia.cloud phi-grafana-01.o11y.eqiad1.wikimedia.cloud phi-kafka-01.o11y.eqiad1.wikimedia.cloud phi-kafkamon-01.o11y.eqiad1.wikimedia.cloud phi-lb-01.o11y.eqiad1.wikimedia.cloud phi-mwlog-01.o11y.eqiad1.wikimedia.cloud phi-pki-01.o11y.eqiad1.wikimedia.cloud phi-prometheus-[01-02].o11y.eqiad1.wikimedia.cloud phi-puppet-01.o11y.eqiad1.wikimedia.cloud phi-syslog-01.o11y.eqiad1.wikimedia.cloud phi-titan-01.o11y.eqiad1.wikimedia.cloud phi-webperf-01.o11y.eqiad1.wikimedia.cloud pixel.pixel.eqiad1.wikimedia.cloud canasta-test.pluggableauth.eqiad1.wikimedia.cloud dcl-dev1.puppet-dev.eqiad1.wikimedia.cloud section-ranker.recommendation-api.eqiad1.wikimedia.cloud semantic-search.recommendation-api.eqiad1.wikimedia.cloud trixie.search.eqiad1.wikimedia.cloud font-db.signwriting.eqiad1.wikimedia.cloud dcl.swift.eqiad1.wikimedia.cloud filippo-tom-k8s-worker-01.testlabs.eqiad1.wikimedia.cloud filippo-tom-pki-01.testlabs.eqiad1.wikimedia.cloud filippo-tom-puppetdb-01.testlabs.eqiad1.wikimedia.cloud pontoon-demo-puppet-01.testlabs.eqiad1.wikimedia.cloud tools-bastion-15.tools.eqiad1.wikimedia.cloud tools-db-7.tools.eqiad1.wikimedia.cloud tools-nfs-3.tools.eqiad1.wikimedia.cloud toolsbeta-nfs-5.toolsbeta.eqiad1.wikimedia.cloud toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud voterlists-1.voterlists.eqiad1.wikimedia.cloud backend.wikicommunityhealth.eqiad1.wikimedia.cloud uwl.wikicommunityhealth.eqiad1.wikimedia.cloud wikibase-metadata.wikidata-dev.eqiad1.wikimedia.cloud wikidata-reconciliation-trixie.wikidata-reconciliation.eqiad1.wikimedia.cloud k3s.wikifunctions.eqiad1.wikimedia.cloud wikipeoplestats-db01.wikipeoplestats.eqiad1.wikimedia.cloud wsexport-app-prod01.wikisource.eqiad1.wikimedia.cloud demo-wiki.wikispeech.eqiad1.wikimedia.cloud glamspore-prod-01.wikispore.eqiad1.wikimedia.cloud ctt-prv-04.wikitextexp.eqiad1.wikimedia.cloud journalist1.wmgmc-monitoring.eqiad1.wikimedia.cloud player1.wmgmc-monitoring.eqiad1.wikimedia.cloud press1.wmgmc-monitoring.eqiad1.wikimedia.cloud xtools-dev08.xtools.eqiad1.wikimedia.cloud xtools-prod[14-15].xtools.eqiad1.wikimedia.cloud zuul-haproxy-01.zuul.eqiad1.wikimedia.cloud zuul-puppetserver-01.zuul.eqiad1.wikimedia.cloud microk8s.zuul3.eqiad1.wikimedia.cloud _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud December 2025