Hi! I'm deploying a tool to Toolforge using the new GitLab CI system, and I got the following error:
gotten_deployment='{"messages":{"info":[],"warning":["You are using a beta feature of Toolforge."],"error":["Unable to find namespace tool-jjpmaster-bot-enwb-t2 or config jjpmaster-bot-enwb-t2-config for jjpmaster-bot-enwb-t2"]}}'The name of the tool is "jjpmaster-bot-enwb-t2" (see https://gitlab.wikimedia.org/toolforge-repos/jjpmaster-bot-enwb-t2/-/jobs/6…). Does anyone know what this error could represent?
Thank you,
JJPMaster
tl;dr:
All cloud-vps instances listed in this email will be rebooted, by me, on
Tuesday December 2nd. This list includes several bastion hosts, where
existing logins, long-running jobs, or tmux sessions will be interrupted.
If you would like more control, you can hard reboot your instances
anytime in the next week and then I will not need to reboot them on the
2nd. Make sure to use the 'Hard reboot instance' menu option; a soft
reboot is not enough to clear the network settings.
details:
We have recently updated our virtualization hardware to slightly
increase the MTU[0], allowing normal-sized packet transmission[1]. This
change was necessary in order to resolve an issue with Docker containers
and, potentially, other issues not yet revealed. Certain VMs that
existed before this update now have an MTU size that doesn't match the
new size set on all hypervisors; this limits our ability to migrate
those VMs and perform other routine maintenance tasks.
A stop/start or 'hard reboot' of affected instances resets the MTU, as
it rebuilds the virtual network stack associated with the host. A 'soft'
reboot (or simply rebooting from within the OS of the instance) leaves
the network stack intact so does not resolve the issue.
If you want to reboot your VMs before Tuesday, there's no need to notify
WMCS staff. I'll re-run the report that lists affected VMs immediately
before rebooting, and a hard reboot will remove your VMs from any future
such report.
-Andrew
[0] https://en.wikipedia.org/wiki/Maximum_transmission_unit
[1] https://phabricator.wikimedia.org/T408543
=====
accounts-appserver7.account-creation-assistance.eqiad1.wikimedia.cloud
deep-dive.analytics.eqiad1.wikimedia.cloud
gitlab-docker-runner-v2.analytics.eqiad1.wikimedia.cloud
T389375.appservers.eqiad1.wikimedia.cloud
rn-hcptchprxy-pki-01.appservers.eqiad1.wikimedia.cloud
rn-hcptchprxy-puppet-01.appservers.eqiad1.wikimedia.cloud
rn-hcptchprxy-puppetdb-01.appservers.eqiad1.wikimedia.cloud
rn-hcptchprxy-urldownloader-[01-02].appservers.eqiad1.wikimedia.cloud
bastion-eqiad1-[5-6].bastion.eqiad1.wikimedia.cloud
k3s.catalyst.eqiad1.wikimedia.cloud
k3s-envDB.catalyst.eqiad1.wikimedia.cloud
k3s-worker[01-02].catalyst.eqiad1.wikimedia.cloud
k3s-worker01.catalyst-dev.eqiad1.wikimedia.cloud
ntp-[5-6].cloudinfra.eqiad1.wikimedia.cloud
mediawiki2latex.collection-alt-renderer.eqiad1.wikimedia.cloud
copypatrol-backend-prod-02.copypatrol.eqiad1.wikimedia.cloud
cvn-apache11.cvn.eqiad1.wikimedia.cloud
cvn-app[13-14].cvn.eqiad1.wikimedia.cloud
deployment-poolcounter07.deployment-prep.eqiad1.wikimedia.cloud
gitlab-1002.devtools.eqiad1.wikimedia.cloud
gitlab-runner-[1007-1008].devtools.eqiad1.wikimedia.cloud
k3s-test.devtools.eqiad1.wikimedia.cloud
generator01.dumpstorrents.eqiad1.wikimedia.cloud
debian13-test.dwl.eqiad1.wikimedia.cloud
taxonbot4.dwl.eqiad1.wikimedia.cloud
tmp.entity-detection.eqiad1.wikimedia.cloud
runner-[1031-1040].gitlab-runners.eqiad1.wikimedia.cloud
wikiwho-dev.globaleducation.eqiad1.wikimedia.cloud
prod0.hashtags.eqiad1.wikimedia.cloud
language-lab.language.eqiad1.wikimedia.cloud
lpl-cx-sx2.language.eqiad1.wikimedia.cloud
lpl-recommend.language.eqiad1.wikimedia.cloud
lpl-services.language.eqiad1.wikimedia.cloud
logging-logstash-04.logging.eqiad1.wikimedia.cloud
mariadbcompiler-trixie.mariadbtest.eqiad1.wikimedia.cloud
ci2.mediawiki-quickstart.eqiad1.wikimedia.cloud
coder-env-1.mobileappsperformance.eqiad1.wikimedia.cloud
wikiapiary.mwstake.eqiad1.wikimedia.cloud
filippo-centrallog-02.o11y.eqiad1.wikimedia.cloud
filippo-cloudcephosd-01.o11y.eqiad1.wikimedia.cloud
filippo-clouddumps-01.o11y.eqiad1.wikimedia.cloud
filippo-cloudgw-01.o11y.eqiad1.wikimedia.cloud
filippo-cloudvirt-[01-02].o11y.eqiad1.wikimedia.cloud
phi-alert-01.o11y.eqiad1.wikimedia.cloud
phi-arclamp-01.o11y.eqiad1.wikimedia.cloud
phi-grafana-01.o11y.eqiad1.wikimedia.cloud
phi-kafka-01.o11y.eqiad1.wikimedia.cloud
phi-kafkamon-01.o11y.eqiad1.wikimedia.cloud
phi-lb-01.o11y.eqiad1.wikimedia.cloud
phi-mwlog-01.o11y.eqiad1.wikimedia.cloud
phi-pki-01.o11y.eqiad1.wikimedia.cloud
phi-prometheus-[01-02].o11y.eqiad1.wikimedia.cloud
phi-puppet-01.o11y.eqiad1.wikimedia.cloud
phi-syslog-01.o11y.eqiad1.wikimedia.cloud
phi-titan-01.o11y.eqiad1.wikimedia.cloud
phi-webperf-01.o11y.eqiad1.wikimedia.cloud
pixel.pixel.eqiad1.wikimedia.cloud
canasta-test.pluggableauth.eqiad1.wikimedia.cloud
dcl-dev1.puppet-dev.eqiad1.wikimedia.cloud
section-ranker.recommendation-api.eqiad1.wikimedia.cloud
semantic-search.recommendation-api.eqiad1.wikimedia.cloud
trixie.search.eqiad1.wikimedia.cloud
font-db.signwriting.eqiad1.wikimedia.cloud
dcl.swift.eqiad1.wikimedia.cloud
filippo-tom-k8s-worker-01.testlabs.eqiad1.wikimedia.cloud
filippo-tom-pki-01.testlabs.eqiad1.wikimedia.cloud
filippo-tom-puppetdb-01.testlabs.eqiad1.wikimedia.cloud
pontoon-demo-puppet-01.testlabs.eqiad1.wikimedia.cloud
tools-bastion-15.tools.eqiad1.wikimedia.cloud
tools-db-7.tools.eqiad1.wikimedia.cloud
tools-nfs-3.tools.eqiad1.wikimedia.cloud
toolsbeta-nfs-5.toolsbeta.eqiad1.wikimedia.cloud
toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud
voterlists-1.voterlists.eqiad1.wikimedia.cloud
backend.wikicommunityhealth.eqiad1.wikimedia.cloud
uwl.wikicommunityhealth.eqiad1.wikimedia.cloud
wikibase-metadata.wikidata-dev.eqiad1.wikimedia.cloud
wikidata-reconciliation-trixie.wikidata-reconciliation.eqiad1.wikimedia.cloud
k3s.wikifunctions.eqiad1.wikimedia.cloud
wikipeoplestats-db01.wikipeoplestats.eqiad1.wikimedia.cloud
wsexport-app-prod01.wikisource.eqiad1.wikimedia.cloud
demo-wiki.wikispeech.eqiad1.wikimedia.cloud
glamspore-prod-01.wikispore.eqiad1.wikimedia.cloud
ctt-prv-04.wikitextexp.eqiad1.wikimedia.cloud
journalist1.wmgmc-monitoring.eqiad1.wikimedia.cloud
player1.wmgmc-monitoring.eqiad1.wikimedia.cloud
press1.wmgmc-monitoring.eqiad1.wikimedia.cloud
xtools-dev08.xtools.eqiad1.wikimedia.cloud
xtools-prod[14-15].xtools.eqiad1.wikimedia.cloud
zuul-haproxy-01.zuul.eqiad1.wikimedia.cloud
zuul-puppetserver-01.zuul.eqiad1.wikimedia.cloud
microk8s.zuul3.eqiad1.wikimedia.cloud
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…
Hello everyone!
We're excited to announce that the next Language Community Meeting is
happening soon - on November 28th at 16:00 UTC! If you’d like to join,
simply sign up on the wiki page
<https://www.mediawiki.org/wiki/Wikimedia_Language_and_Product_Localization/…>
.
This is a participant-driven meeting where we share updates on
language-related projects, discuss technical challenges in language wikis,
and collaborate on solutions. For example, in our upcoming meeting, we plan
to hear from contributors of the Wikitongues project and Fante
Wikimedia Community.
Got a topic to share? Whether it’s a technical update from your project, a
challenge you need help with, or a request for interpretation support, we’d
love to hear from you! Feel free to reply to this message or add agenda
items to the document here
<https://etherpad.wikimedia.org/p/language-community-meeting-nov-2025>.
Also, we’d like to highlight that the 9th edition of the Language &
Internationalization Newsletter (October 2025)
<https://www.mediawiki.org/wiki/Wikimedia_Language_and_Product_Localization/…>
is
now available. This newsletter provides updates from the July–September
2025 quarter on new feature development, improvements in various
language-related technical projects and support efforts, details about
community meetings, and ideas for contributing to projects. To stay
updated, you can subscribe to the newsletter
<https://www.mediawiki.org/wiki/Newsletter:Language_and_Internationalization…>
on
its wiki page.
Are you interested in contributing to the technical work around language
development? See a curated list of technical contribution tasks here:
T407935 <https://phabricator.wikimedia.org/T407935>.
We look forward to your ideas and participation at the Language Community
Meeting. See you there!
Cheers,
Srishti
*Srishti Sethi*
Senior Developer Advocate
Wikimedia Foundation <https://wikimediafoundation.org/>
Hi!
I see in the documentation
<https://wikitech.wikimedia.org/wiki/PAWS/Python_with_Pip> that it is
possible to install additional Python libraries in PAWS using pip like so:
import sys
!{sys.executable} -m pip install [PACKAGE]
I see libraries are installed to /srv/paws/lib/python3.12/site-packages. I
assume this is not shared across PAWS users. By the way, does this count
toward the 5 GB storage available?
Is there a recommended way to create and use Python virtual environments in
PAWS notebooks? I think that would make it easier to identify which
packages and dependencies have been installed for which notebooks, and to
remove them when no longer needed.
The procedure I use for my local notebooks is the following (as explained
here <https://ipython.readthedocs.io/en/8.21.0/install/kernel_install.html>),
but I'm not sure this would be appropriate for PAWS JupyterLab setup:
1. Create a virtual environment: python -m venv venv
2. Activate the virtual environment: source venv/bin/activate
3. Install ipykernel on the virtual environment: pip install ipykernel
4. Configure the virtual environment's ipykernel into user's Jupyter
configuration: python -m ipykernel install --user --name=venv-name
TL;DR:
- You can make an HTTP redirector using the new
tool-containers/redirect:latest [0]
In 2020 I documented a way to use a bare Kubernetes ingress object to
redirect HTTP requests from one tool to another.[1] This method was
used by a number of tools I managed and some additional tools to give
aliases to help with discoverability, redirect to wiki pages, redirect
to Cloud VPS projects, or migrate users to a different tool. It was a
nice solution at the time because it did not require additional
resources from the Kubernetes cluster beyond configuring the existing
shared nginx ingress service.
Changes in newer versions of the nginx ingress product have broken the
prior method, so we needed to find a different solution. I spent a few
days poking at the problem and decided to write a small golang
webserver that narrowly does the things that were needed to replace
the existing redirecting ingress objects.[0][2] I have packaged this
as a Toolforge build service managed container that any Toolforge tool
can pull from the registry for its own use.
Taavi and I have replaced all of the legacy redirecting ingress
objects we could find with the new containerized solution.[3] If you
have a tool we missed and need help converting, please do reach out to
me via email, irc, phabricator, or carrier pigeon.
If you have a tool that is doing redirects a different way, like with
lighttpd config and a legacy PHP container, take a look at the new
service and see if it might be a good replacement. Feature requests
can be filed in Phabricator as well.[4]
[0]: https://wikitech.wikimedia.org/wiki/Tool:Containers#Redirect_container
[1]: https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_…
[2]: https://gitlab.wikimedia.org/toolforge-repos/containers-redirect
[3]: https://phabricator.wikimedia.org/T409474
[4]: https://phabricator.wikimedia.org/tag/tool-containers/
Bryan
--
Bryan Davis Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis (WMF)]] irc: bd808
I will be rolling out some network setting changes on the Cloud VPS
hypervisors and other networking equipment over the course of today.
Applying the changes on a hypervisor will cause network interruptions
for roughly 5-10 seconds for all VMs hosted on it.
For technical details, see [0].
[0]: https://phabricator.wikimedia.org/T330075
Taavi
--
Taavi Väänänen (he/they)
Site Reliability Engineer, Tools Infrastructure
Wikimedia Foundation
_______________________________________________
Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org
List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…