Reminder: this upgrade is coming up in about an hour.
On 8/17/26 2:12 PM, Andrew Bogott wrote:
> Next Wednesday we will up upgrading the Toolforge Kubernetes install
> to version 1.33.
>
> This upgrade should not result in any feature changes or downtime but
> there may be brief network resets and pods may be restarted and/or
> rescheduled.
>
> This upgrade is tracked here: https://phabricator.wikimedia.org/T433132
>
> -Andrew
>
Hello,
This is a friendly reminder that the next Toolforge Monthly Meeting is
coming up on Tuesday, August 25.
The Toolforge monthly meeting is a one-hour engineering-focused meeting to
discuss the current state and the future plans of Toolforge, to unblock
some decisions or to get everyone on the same page. It's also an
opportunity for technical contributors to connect with the Tools Platform /
Infrastructure teams and contribute to the ongoing and future work to
improve Toolforge.
The meeting agenda and joining information are available on the meeting
wiki page[0].
The current agenda includes:
-
Kubernetes upgrade workgroup progress
-
Push-to-Deploy updates
If there are other topics you'd like to discuss, please feel free to add it
to the agenda on the wiki before the meeting. We'd be happy to include
community-driven discussion topics.
Whether you're a long-time Toolforge maintainer or simply interested in the
platform, you're welcome to join the discussion.
We look forward to seeing you there!
[0]:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Monthly_meeting
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
Hi all,
The Toolforge Elasticsearch cluster is being replaced with a cluster
based on OpenSearch v2[0]. If your tool uses the writable
Elastic/OpenSearch functionality you will need to manually migrate
your tool to the new cluster as described in the rest of this email.
(Note that this is different from the "Cloud Elastic" service which
provides read-only access to the search indexes of Wikimedia wikis.)
The Wikitech page[0] has been updated with the details of the new
cluster, which are almost identical to the previous cluster except the
hostname and software version used. The same set of credentials will
also work on the new cluster. There are, however, some breaking API
changes between the old version and the new one, which is why we are
not just migrating all indexes for all tools at once.
To help migrate data from the old cluster to the new one, we have
temporarily added a `toolforge migrate-index` command to the bastion
servers to copy over an index and its mappings, aliases and
replication settings. The tool takes a space-separated list of indexes
to migrate, and will use credentials saved as envvars by default (and
prompt you if those are not available). Migration is relatively quick,
in my tests an index with about 400k records and 1+1 replication takes
about a minute to migrate.
The old Elasticsearch cluster will be shut down on 24 August.
This work is tracked as T401818.[1]
[0]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/OpenSearch
[1]: https://phabricator.wikimedia.org/T401818
--
Taavi Väänänen (he/they)
Site Reliability Engineer, Tools Infrastructure
Wikimedia Foundation
Hello,
This is a reminder that Debian Bullseye will reach the end of its Long Term
Support (LTS) on August 31, 2026[0]
If your Cloud VPS project still has instances running Debian Bullseye, now
is the time to migrate them to **Debian Bookworm or Debian Trixie**.
We have published a migration guide to help with the process.[1]
As part of this migration, it may also be worth evaluating whether your
workload is a good fit for Toolforge.
If feasible, migrating to Toolforge can reduce future maintenance burden,
as operating system upgrades are much simpler.
Our goal is to complete these migrations before Bullseye reaches end of
support. After the upstream LTS deadline,
Bullseye instances may become subject to shutdown or cleanup as part of the
deprecation process.
If you are unable to complete your migration before August 31, or need
assistance, please let us know as soon as possible.
In addition to replying on this mailing list or contacting the Tools
Platform team[2], please file a Phabricator task
tagged **Cloud-VPS (Debian Bullseye Deprecation[3])** so we can track your
project and work with you on a migration plan.
[0]: https://phabricator.wikimedia.org/T401804
[1]:
https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_instance_operating_syste…
[2]:
https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS#Communication_and_support
[3]: https://phabricator.wikimedia.org/project/view/8107/
Thanks!
--
Seyram Komla Sapaty
Developer Advocate
Wikimedia Cloud Services
Next Wednesday we will up upgrading the Toolforge Kubernetes install to
version 1.33.
This upgrade should not result in any feature changes or downtime but
there may be brief network resets and pods may be restarted and/or
rescheduled.
This upgrade is tracked here: https://phabricator.wikimedia.org/T433132
-Andrew
(If your third-party tool does not read data from [wbc_entity_usage] via
Wiki Replicas, please ignore this email)
Hello everyone,
planning is underway to move the Wikibase [wbc_entity_usage] tables for all
wikis to a different “*x1 extension cluster*” to improve database space
usage [0]. We expect that few, (if any) third-party tools directly rely on
this table — but before making any breaking changes, we want to confirm
that.
If your tool uses this table, hearing from you will help us understand how
many tools are affected and what that usage looks like, so we can plan the
migration accordingly.
[wbc_entity_usage] tracks which Wikidata entities are used on a given
Wiki’s pages, and also powers Wikidata-triggered Recent Changes and
Watchlist injections to client Wikis.
*// What's affected //*
If you operate a tool that reads from the [wbc_entity_usage] table —
through Wiki Replicas or directly from SQL, this may affect you.
This is especially likely if your query JOINs [wbc_entity_usage] with other
MediaWiki tables (e.g. ‘page’, ‘revision’), since cross-cluster JOINs will
stop working when the migration is completed.
*// What to do //*
Please comment on [1] (even a simple “yes, I am affected” is helpful). If
you can share more, we'd like to know:
-
The Tool name and location (e.g. Toolforge, repo link).
-
If your query JOIN’s [wbc_entity_usage] with other tables, please tell
us which.
-
How often the queries run against [wbc_entity_usage], (e.g. real-time,
daily, weekly, manually).
Please be aware:
-
[wbc_entity_usage] will eventually be removed from Wiki Replicas and
direct SQL. Existing queries against this table will need to be replaced
with a different approach.
-
During the lengthy transition period, the tables will still operate, so
existing queries will continue to function until access is turned off when
the migration is complete [2].
-
Once the Wiki Replicas access ends, tool users should switch over to
using the MediaWiki API [3] or Recent Changes API [4] to read changes
instead.
Queries on the table will no longer work *after* the migration is
completed, so they will need to be adjusted.
*// Links //*
[0] - https://phabricator.wikimedia.org/T424133
[2] - https://phabricator.wikimedia.org/T424440
[3] -
https://en.wikipedia.org/w/api.php?action=help&modules=query%2Bwbentityusage
[4] - https://www.mediawiki.org/wiki/API:RecentChanges
Please report if your tool is affected to the relevant Phabricator ticket:
[1] - https://phabricator.wikimedia.org/T430289
Thank you,
Danny (on behalf of the Wikidata For Wikimedia Projects
<https://meta.wikimedia.org/wiki/Wikidata_For_Wikimedia_Projects> team)
--
*Danny Benjafield*
Community Communications Manager
Wikidata For Wikimedia Projects
Wikimedia Deutschland e. V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Phone: +49 (0)30-577 11 62-0
https://wikimedia.de
Keep up to date! Current news and exciting stories about Wikimedia,
Wikipedia and Free Knowledge in our newsletter (in German): Subscribe now.
<https://www.wikimedia.de/newsletter/>
Imagine a world in which every single human being can freely share in the
sum of all knowledge. Help us to achieve our vision!
https://spenden.wikimedia.de
Wikimedia Deutschland — Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Charlottenburg, VR 23855 B.
Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin,
Steuernummer 27/029/42207. Geschäftsführende Vorstände: Franziska Heine
This email relates to the shared /data/scratch mount used by toolforge
and some cloud-vps projects. It is provided to the following projects:
- dumps
- language
- maps
- math
- mwoffliner
- paws
- statanalyser
- testlabs
- tools
- toolsbeta
- video
- wikidata-query
- wikidumpparse
- wikiqlever
As part of our push to eliminate the use of Debian Bullseye, we will be
replacing the server that hosts the nfs scratch volume on Monday,
sometime around 15:00 UTC. Testing suggests that this will only produce
a few minutes of downtime for the volume (likely reported as 'Permission
Denied'); if you find your system frozen please give it a few minutes
and then reboot if waiting doesn't help.
The data on the old server will be transferred to the new server, so no
data will be lost.[0]
Bonus points: If you are a cloud-vps admin and have read this far,
please consider whether you still need the scratch volume. This service
was initially provided before we had attachable block storage
available[1], so many users rely on scratch not for file transfer
between hosts but because they needed extra storage space. If you are
one of those cases, please open a quota ticket[2] and mention that you
are moving off of scratch. Life without NFS is better for everyone!
This work is tracked on phabricator here:
https://phabricator.wikimedia.org/T433426
- Andrew + the WMCS team
[0] ...but if you have data on scratch that you care about, it should be
somewhere else! We make no reliability guarantees for this storage.
[1]
https://wikitech.wikimedia.org/wiki/Help:Adding_disk_space_to_Cloud_VPS_ins…
[2] https://phabricator.wikimedia.org/project/view/2880/