Cloud

cloud@lists.wikimedia.org

2 participants
949 discussions

[Cloud-announce] Toolforge Kubernetes maintenance on 2026-01-14
by Taavi Väänänen 07 Jan '26

07 Jan '26

Hi, We will be upgrading the Toolforge Kubernetes cluster[0] on January 14th (next Wednesday) starting at around 14:00 UTC. The expected impact is that all tool webservices and running jobs will get restarted a couple of times over the course of the few hours it takes for us to upgrade the entire cluster. The ability to manage tools will remain operational. [0]: https://phabricator.wikimedia.org/T413797 Taavi -- Taavi Väänänen (he/they) Site Reliability Engineer, Tools Infrastructure Wikimedia Foundation _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

[Cloud-announce] New Toolforge Standards Committee members appointed
by JJMC89 05 Jan '26

05 Jan '26

Following a public nomination process[0], request for feedback[1], and signing of non-disclosure agreements with the Wikimedia Foundation[2] the following users have been appointed as the newest incarnation of the Toolforge Standards Committee[3]: * JJMC89 * Lucas Werkmeister * Pintoch * SD0001 * Sohom Datta They are now empowered to help resolve requests related to the Right to fork policy[4] and the Abandoned tool policy[5]. They may also choose as a group to promote other standards and practices within Toolforge that are intended to help us all make and support better tools for the community. [0]: https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.… [1]: https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/messa… [2]: https://phabricator.wikimedia.org/T413434 [3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Toolforge_standards_comm… [4]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Right_to_fork_policy [5]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 1

502 Bad Gateway error in public-paws.wmcloud.org
by delahera＠gmail.com 23 Dec '25

23 Dec '25

Hi all! I'm working with some notebooks on PAWS and sometimes I get a 502 Bad Gateway error when trying to access their HTML version via public-paws.wmcloud.org. See for example https://public-paws.wmcloud.org/User:Diegodlh/wmuy-caad/semantic.ipynb. You should still be able to get the raw version appending `?format=raw` at the end. I think it may have something to do with the notebook's output or metadata, since the problem seems to go away when clearing the outputs (but usually reappears after running the notebook again). I thought I had identified some cells causing the issue, so I added some code to clear the output after running (look for `clear_output()` in the raw notebook above). This seemed to work some time, but it's failing again now. My understanding about how PAWS works internally is very basic and I don't know what else I can do to debug this. I searched for this problem in Phabricator, this mailing list and the Telegram channel, but didn't find anything. Could someone help me see what may be going on and how to fix it? Thanks!

1 0

Resource consumption on toolforge
by Nicolas Vervelle 07 Dec '25

07 Dec '25

Hi everyone, I'm seeing strange things with resource consumption with toolforge. My resource consumption reported by the quota is way above the consumption of my currently running job, preventing other jobs from starting. Where does the extra consumption come from ? And how to clean this ? More detailed data below Any help appreciated, NicoV My tool account is wpcleaner on toolforge. Only one job is currently running : tools.wpcleaner@tools-bastion-15:~$ toolforge jobs list +------------------------+--------------------+------------------------------------------+ | Job name: | Job type: | Status: | +------------------------+--------------------+------------------------------------------+ | wpcleaner-cs-weekly | schedule: @weekly | Last schedule time: 2025-12-03T05:17:00Z | | wpcleaner-en-list | schedule: @weekly | Unable to start, out of quota for memory | | wpcleaner-fr-dab | schedule: @weekly | Unable to start, out of quota for memory | | wpcleaner-fr-daily | schedule: @daily | Last schedule time: 2025-12-07T03:58:00Z | | wpcleaner-fr-list | schedule: @weekly | Running for 2d4h10m | | wpcleaner-fr-weekly | schedule: @weekly | Last schedule time: 2025-12-04T05:03:00Z | | wpcleaner-meta-list | schedule: @monthly | Last schedule time: 2025-12-04T04:25:00Z | | wpcleaner-meta-monthly | schedule: @monthly | Last schedule time: 2025-11-18T06:05:00Z | +------------------------+--------------------+------------------------------------------+ The currently running job defines 3G and 1 CPU for resources : tools.wpcleaner@tools-bastion-15:~$ toolforge jobs show wpcleaner-fr-list +---------------+-----------------------------------------------------------------+ | Job name: | wpcleaner-fr-list | +---------------+-----------------------------------------------------------------+ | Command: | /data/project/wpcleaner/tools/scripts/fr_ListCheckWiki_List.sh | +---------------+-----------------------------------------------------------------+ | Job type: | schedule: @weekly | +---------------+-----------------------------------------------------------------+ | Image: | jdk17 | +---------------+-----------------------------------------------------------------+ | Port: | none | +---------------+-----------------------------------------------------------------+ | File log: | no | +---------------+-----------------------------------------------------------------+ | Output log: | | +---------------+-----------------------------------------------------------------+ | Error log: | | +---------------+-----------------------------------------------------------------+ | Emails: | onfailure | +---------------+-----------------------------------------------------------------+ | Resources: | mem: 3.0Gi, cpu: 1.0 | +---------------+-----------------------------------------------------------------+ | Replicas: | | +---------------+-----------------------------------------------------------------+ | Mounts: | none | +---------------+-----------------------------------------------------------------+ | Retry: | no | +---------------+-----------------------------------------------------------------+ | Timeout: | no | +---------------+-----------------------------------------------------------------+ | Health check: | none | +---------------+-----------------------------------------------------------------+ | Status: | Running for 2d4h11m | +---------------+-----------------------------------------------------------------+ | Hints: | Last run at 2025-12-03T20:36:14Z. Pod in 'Running' phase. State | | | 'running'. Started at '2025-12-03T20:36:15Z'. | +---------------+-----------------------------------------------------------------+ But the quota command says I'm consuming 2.5 CPU and 6.5G, which is a lot more than what the only running job defines. tools.wpcleaner@tools-bastion-15:~$ toolforge jobs quota Running jobs Used Limit -------------------------------------------- ------ ------- Total running jobs at once (Kubernetes pods) 3 16 Running one-off and cron jobs 4 15 CPU 2.5 16.0 Memory 6.5Gi 8.0Gi Per-job limits Used Limit ---------------- ------ ------- CPU 3.0 Memory 6.0Gi Job definitions Used Limit ---------------------------------------- ------ ------- Cron jobs 8 50 Continuous jobs (including web services) 1 16

2 3

[Cloud-announce] Toolforge standards committee nominations open
by JJ Clark 01 Dec '25

01 Dec '25

TLDR: Nominations are open for the Toolforge standards committee[0] at https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Toolforge_standards… Following the TFSC's revision to the appointment process[1] from appointment as needed by the Foundation to an annual appointment by the existing committee, this is the first annual appointment nomination period. Nominations are being collected on Wikitech[2] through 2025-11-28. The committee's primary duties are to oversee the Right to fork policy[3] and Abandoned tool policy[4]. Any Toolforge project member in good standing is eligible for nomination. Self-nominations are acceptable. Nominations by a third party must be counter-signed by the nominee to indicate they would accept the responsibility and be willing to sign the Volunteer NDA[5]. Following the nomination period there will be a two-week period for the Toolforge admins and community to provide feedback on the nominees to the current committee. Following the feedback period, the current committee will vote on the nominees. Nominees approved by the committee must sign the Volunteer NDA to be appointed to the committee. [0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Toolforge_standards_comm… [1]: https://wikitech.wikimedia.org/wiki/Special:Diff/2357725 [2]: https://wikitech.wikimedia.org/wiki/Help_talk:Toolforge/Toolforge_standards… [3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Right_to_fork_policy [4]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Abandoned_tool_policy [5]: https://wikitech.wikimedia.org/wiki/Volunteer_NDA _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 1

GitLab/Toolforge deployment error
by jjpmaster 01 Dec '25

01 Dec '25

Hi! I'm deploying a tool to Toolforge using the new GitLab CI system, and I got the following error: gotten_deployment='{"messages":{"info":[],"warning":["You are using a beta feature of Toolforge."],"error":["Unable to find namespace tool-jjpmaster-bot-enwb-t2 or config jjpmaster-bot-enwb-t2-config for jjpmaster-bot-enwb-t2"]}}'The name of the tool is "jjpmaster-bot-enwb-t2" (see https://gitlab.wikimedia.org/toolforge-repos/jjpmaster-bot-enwb-t2/-/jobs/6…). Does anyone know what this error could represent? Thank you, JJPMaster

2 2

[Cloud-announce] Reboot (some of) your instances, or expect them to be rebooted for you on December 2
by Andrew Bogott 01 Dec '25

01 Dec '25

tl;dr: All cloud-vps instances listed in this email will be rebooted, by me, on Tuesday December 2nd. This list includes several bastion hosts, where existing logins, long-running jobs, or tmux sessions will be interrupted. If you would like more control, you can hard reboot your instances anytime in the next week and then I will not need to reboot them on the 2nd. Make sure to use the 'Hard reboot instance' menu option; a soft reboot is not enough to clear the network settings. details: We have recently updated our virtualization hardware to slightly increase the MTU[0], allowing normal-sized packet transmission[1]. This change was necessary in order to resolve an issue with Docker containers and, potentially, other issues not yet revealed. Certain VMs that existed before this update now have an MTU size that doesn't match the new size set on all hypervisors; this limits our ability to migrate those VMs and perform other routine maintenance tasks. A stop/start or 'hard reboot' of affected instances resets the MTU, as it rebuilds the virtual network stack associated with the host. A 'soft' reboot (or simply rebooting from within the OS of the instance) leaves the network stack intact so does not resolve the issue. If you want to reboot your VMs before Tuesday, there's no need to notify WMCS staff. I'll re-run the report that lists affected VMs immediately before rebooting, and a hard reboot will remove your VMs from any future such report. -Andrew [0] https://en.wikipedia.org/wiki/Maximum_transmission_unit [1] https://phabricator.wikimedia.org/T408543 ===== accounts-appserver7.account-creation-assistance.eqiad1.wikimedia.cloud deep-dive.analytics.eqiad1.wikimedia.cloud gitlab-docker-runner-v2.analytics.eqiad1.wikimedia.cloud T389375.appservers.eqiad1.wikimedia.cloud rn-hcptchprxy-pki-01.appservers.eqiad1.wikimedia.cloud rn-hcptchprxy-puppet-01.appservers.eqiad1.wikimedia.cloud rn-hcptchprxy-puppetdb-01.appservers.eqiad1.wikimedia.cloud rn-hcptchprxy-urldownloader-[01-02].appservers.eqiad1.wikimedia.cloud bastion-eqiad1-[5-6].bastion.eqiad1.wikimedia.cloud k3s.catalyst.eqiad1.wikimedia.cloud k3s-envDB.catalyst.eqiad1.wikimedia.cloud k3s-worker[01-02].catalyst.eqiad1.wikimedia.cloud k3s-worker01.catalyst-dev.eqiad1.wikimedia.cloud ntp-[5-6].cloudinfra.eqiad1.wikimedia.cloud mediawiki2latex.collection-alt-renderer.eqiad1.wikimedia.cloud copypatrol-backend-prod-02.copypatrol.eqiad1.wikimedia.cloud cvn-apache11.cvn.eqiad1.wikimedia.cloud cvn-app[13-14].cvn.eqiad1.wikimedia.cloud deployment-poolcounter07.deployment-prep.eqiad1.wikimedia.cloud gitlab-1002.devtools.eqiad1.wikimedia.cloud gitlab-runner-[1007-1008].devtools.eqiad1.wikimedia.cloud k3s-test.devtools.eqiad1.wikimedia.cloud generator01.dumpstorrents.eqiad1.wikimedia.cloud debian13-test.dwl.eqiad1.wikimedia.cloud taxonbot4.dwl.eqiad1.wikimedia.cloud tmp.entity-detection.eqiad1.wikimedia.cloud runner-[1031-1040].gitlab-runners.eqiad1.wikimedia.cloud wikiwho-dev.globaleducation.eqiad1.wikimedia.cloud prod0.hashtags.eqiad1.wikimedia.cloud language-lab.language.eqiad1.wikimedia.cloud lpl-cx-sx2.language.eqiad1.wikimedia.cloud lpl-recommend.language.eqiad1.wikimedia.cloud lpl-services.language.eqiad1.wikimedia.cloud logging-logstash-04.logging.eqiad1.wikimedia.cloud mariadbcompiler-trixie.mariadbtest.eqiad1.wikimedia.cloud ci2.mediawiki-quickstart.eqiad1.wikimedia.cloud coder-env-1.mobileappsperformance.eqiad1.wikimedia.cloud wikiapiary.mwstake.eqiad1.wikimedia.cloud filippo-centrallog-02.o11y.eqiad1.wikimedia.cloud filippo-cloudcephosd-01.o11y.eqiad1.wikimedia.cloud filippo-clouddumps-01.o11y.eqiad1.wikimedia.cloud filippo-cloudgw-01.o11y.eqiad1.wikimedia.cloud filippo-cloudvirt-[01-02].o11y.eqiad1.wikimedia.cloud phi-alert-01.o11y.eqiad1.wikimedia.cloud phi-arclamp-01.o11y.eqiad1.wikimedia.cloud phi-grafana-01.o11y.eqiad1.wikimedia.cloud phi-kafka-01.o11y.eqiad1.wikimedia.cloud phi-kafkamon-01.o11y.eqiad1.wikimedia.cloud phi-lb-01.o11y.eqiad1.wikimedia.cloud phi-mwlog-01.o11y.eqiad1.wikimedia.cloud phi-pki-01.o11y.eqiad1.wikimedia.cloud phi-prometheus-[01-02].o11y.eqiad1.wikimedia.cloud phi-puppet-01.o11y.eqiad1.wikimedia.cloud phi-syslog-01.o11y.eqiad1.wikimedia.cloud phi-titan-01.o11y.eqiad1.wikimedia.cloud phi-webperf-01.o11y.eqiad1.wikimedia.cloud pixel.pixel.eqiad1.wikimedia.cloud canasta-test.pluggableauth.eqiad1.wikimedia.cloud dcl-dev1.puppet-dev.eqiad1.wikimedia.cloud section-ranker.recommendation-api.eqiad1.wikimedia.cloud semantic-search.recommendation-api.eqiad1.wikimedia.cloud trixie.search.eqiad1.wikimedia.cloud font-db.signwriting.eqiad1.wikimedia.cloud dcl.swift.eqiad1.wikimedia.cloud filippo-tom-k8s-worker-01.testlabs.eqiad1.wikimedia.cloud filippo-tom-pki-01.testlabs.eqiad1.wikimedia.cloud filippo-tom-puppetdb-01.testlabs.eqiad1.wikimedia.cloud pontoon-demo-puppet-01.testlabs.eqiad1.wikimedia.cloud tools-bastion-15.tools.eqiad1.wikimedia.cloud tools-db-7.tools.eqiad1.wikimedia.cloud tools-nfs-3.tools.eqiad1.wikimedia.cloud toolsbeta-nfs-5.toolsbeta.eqiad1.wikimedia.cloud toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud voterlists-1.voterlists.eqiad1.wikimedia.cloud backend.wikicommunityhealth.eqiad1.wikimedia.cloud uwl.wikicommunityhealth.eqiad1.wikimedia.cloud wikibase-metadata.wikidata-dev.eqiad1.wikimedia.cloud wikidata-reconciliation-trixie.wikidata-reconciliation.eqiad1.wikimedia.cloud k3s.wikifunctions.eqiad1.wikimedia.cloud wikipeoplestats-db01.wikipeoplestats.eqiad1.wikimedia.cloud wsexport-app-prod01.wikisource.eqiad1.wikimedia.cloud demo-wiki.wikispeech.eqiad1.wikimedia.cloud glamspore-prod-01.wikispore.eqiad1.wikimedia.cloud ctt-prv-04.wikitextexp.eqiad1.wikimedia.cloud journalist1.wmgmc-monitoring.eqiad1.wikimedia.cloud player1.wmgmc-monitoring.eqiad1.wikimedia.cloud press1.wmgmc-monitoring.eqiad1.wikimedia.cloud xtools-dev08.xtools.eqiad1.wikimedia.cloud xtools-prod[14-15].xtools.eqiad1.wikimedia.cloud zuul-haproxy-01.zuul.eqiad1.wikimedia.cloud zuul-puppetserver-01.zuul.eqiad1.wikimedia.cloud microk8s.zuul3.eqiad1.wikimedia.cloud _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

Upcoming Language Community Meeting (November 28th, 16:00 UTC), Newsletter, and Technical Contribution Tasks
by Srishti Sethi 27 Nov '25

27 Nov '25

Hello everyone! We're excited to announce that the next Language Community Meeting is happening soon - on November 28th at 16:00 UTC! If you’d like to join, simply sign up on the wiki page <https://www.mediawiki.org/wiki/Wikimedia_Language_and_Product_Localization/…> . This is a participant-driven meeting where we share updates on language-related projects, discuss technical challenges in language wikis, and collaborate on solutions. For example, in our upcoming meeting, we plan to hear from contributors of the Wikitongues project and Fante Wikimedia Community. Got a topic to share? Whether it’s a technical update from your project, a challenge you need help with, or a request for interpretation support, we’d love to hear from you! Feel free to reply to this message or add agenda items to the document here <https://etherpad.wikimedia.org/p/language-community-meeting-nov-2025>. Also, we’d like to highlight that the 9th edition of the Language & Internationalization Newsletter (October 2025) <https://www.mediawiki.org/wiki/Wikimedia_Language_and_Product_Localization/…> is now available. This newsletter provides updates from the July–September 2025 quarter on new feature development, improvements in various language-related technical projects and support efforts, details about community meetings, and ideas for contributing to projects. To stay updated, you can subscribe to the newsletter <https://www.mediawiki.org/wiki/Newsletter:Language_and_Internationalization…> on its wiki page. Are you interested in contributing to the technical work around language development? See a curated list of technical contribution tasks here: T407935 <https://phabricator.wikimedia.org/T407935>. We look forward to your ideas and participation at the Language Community Meeting. See you there! Cheers, Srishti *Srishti Sethi* Senior Developer Advocate Wikimedia Foundation <https://wikimediafoundation.org/>

1 2

Python virtual environments in PAWS
by Diego de la Hera 20 Nov '25

20 Nov '25

Hi! I see in the documentation <https://wikitech.wikimedia.org/wiki/PAWS/Python_with_Pip> that it is possible to install additional Python libraries in PAWS using pip like so: import sys !{sys.executable} -m pip install [PACKAGE] I see libraries are installed to /srv/paws/lib/python3.12/site-packages. I assume this is not shared across PAWS users. By the way, does this count toward the 5 GB storage available? Is there a recommended way to create and use Python virtual environments in PAWS notebooks? I think that would make it easier to identify which packages and dependencies have been installed for which notebooks, and to remove them when no longer needed. The procedure I use for my local notebooks is the following (as explained here <https://ipython.readthedocs.io/en/8.21.0/install/kernel_install.html>), but I'm not sure this would be appropriate for PAWS JupyterLab setup: 1. Create a virtual environment: python -m venv venv 2. Activate the virtual environment: source venv/bin/activate 3. Install ipykernel on the virtual environment: pip install ipykernel 4. Configure the virtual environment's ipykernel into user's Jupyter configuration: python -m ipykernel install --user --name=venv-name

2 2

New method for redirecting a Toolforge tool to another website
by Bryan Davis 19 Nov '25

19 Nov '25

TL;DR: - You can make an HTTP redirector using the new tool-containers/redirect:latest [0] In 2020 I documented a way to use a bare Kubernetes ingress object to redirect HTTP requests from one tool to another.[1] This method was used by a number of tools I managed and some additional tools to give aliases to help with discoverability, redirect to wiki pages, redirect to Cloud VPS projects, or migrate users to a different tool. It was a nice solution at the time because it did not require additional resources from the Kubernetes cluster beyond configuring the existing shared nginx ingress service. Changes in newer versions of the nginx ingress product have broken the prior method, so we needed to find a different solution. I spent a few days poking at the problem and decided to write a small golang webserver that narrowly does the things that were needed to replace the existing redirecting ingress objects.[0][2] I have packaged this as a Toolforge build service managed container that any Toolforge tool can pull from the registry for its own use. Taavi and I have replaced all of the legacy redirecting ingress objects we could find with the new containerized solution.[3] If you have a tool we missed and need help converting, please do reach out to me via email, irc, phabricator, or carrier pigeon. If you have a tool that is doing redirects a different way, like with lighttpd config and a legacy PHP container, take a look at the new service and see if it might be a good replacement. Feature requests can be filed in Phabricator as well.[4] [0]: https://wikitech.wikimedia.org/wiki/Tool:Containers#Redirect_container [1]: https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_… [2]: https://gitlab.wikimedia.org/toolforge-repos/containers-redirect [3]: https://phabricator.wikimedia.org/T409474 [4]: https://phabricator.wikimedia.org/tag/tool-containers/ Bryan -- Bryan Davis Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis (WMF)]] irc: bd808

1 0

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud