Hi,
We will be upgrading the Toolforge Kubernetes cluster[0] on June 15th
(next Monday) starting at around 11:00 UTC[1].
The expected impact is that all tool webservices and running jobs will
get restarted a couple of times over the course of the few hours it
takes for us to upgrade the entire cluster. The ability to manage
tools will remain operational.
[0]: https://phabricator.wikimedia.org/T427919
[1]: https://zonestamp.toolforge.org/1781521200
taavi
--
Taavi Väänänen (he/they)
Site Reliability Engineer, Tools Infrastructure
Wikimedia Foundation
Hi,
We will be upgrading[0] the Toolforge Kubernetes component responsible
for validating workloads running on the cluster (Kyverno) on June 9th
(next Tuesday) starting at around 11:00 UTC[1].
No downtime is expected, although past Kyverno upgrades have a history
of being bumpier than upgrades of other similar infrastructure
components, so there is a minor risk of unexpected downtime for
starting new jobs.
[0]: https://phabricator.wikimedia.org/T427928
[1]: https://zonestamp.toolforge.org/1781002800
taavi
--
Taavi Väänänen (he/they)
Site Reliability Engineer, Tools Infrastructure
Wikimedia Foundation
The Toolforge Jobs Framework recently switched to its own storage layer [0]
for tracking jobs,
but we temporarily retained the ability to reconcile jobs created directly
via k8s CLIs/APIs
with Toolforge.
This temporary measure will be removed on June 20, 2026. After that date,
"toolforge jobs"
commands will only see jobs created through the Toolforge CLI or API. Jobs
created via k8s
CLIs/APIs will continue to run as Kubernetes objects — they just won't show
in "toolforge jobs"
output.
If you mix k8s CLIs/APIs and "toolforge jobs" to manage your workloads,
please refactor your
workflows to use "toolforge jobs" CLI/API instead of k8s CLIs/APIs before
June 20.
If you only use "toolforge jobs" commands, no action is needed.
For questions, reach out on IRC (#wikimedia-cloud), this list, or the Tools
Platform Team
phabricator tag [1].
[0] https://phabricator.wikimedia.org/T359650
[1] https://phabricator.wikimedia.org/tag/tools-platform-team/
Hello everyone,
As part of the ongoing upgrade of our Search infrastructure from OpenSearch
1.3 → 2.19 → 3.5, we recently upgraded the first production cluster (
cloudelastic).
Following the upgrade, the cluster has become unstable. In particular, some
indices have entered a read-only (red) state, which may impact search
functionality relying on this cluster.
We are currently investigating the issue from multiple angles. Early
indications suggest this may be related to readahead settings in
combination with the underlying storage hardware, but this is not yet
confirmed.
Tracking task: https://phabricator.wikimedia.org/T424852
Our current focus is on stabilizing the cluster and restoring full
functionality. We will share updates as we learn more. At this point, we
expect to have the cluster stabilized again by the end of the week.
If you are observing issues that you suspect are related, please add
details to the Phabricator task or reach out to the Search team.
Thanks for your patience,
Peter
--
Peter Fischer (he/him)
Senior Software Engineer, Search Platform
Wikimedia Foundation
Hello all,
in order to optimize Toolforge resources utilization we will be
progressively changing the default memory requests for webservice tools
from 256MB to 64MB. Memory and CPU requests are scheduler hints for
Kubernetes on how to allocate tools to worker nodes; the default memory
limit (i.e. enforced maximum memory usage) stays unchanged at 512MB. There
is no expected impact to tools using more memory than they request.
The first phase will involve changing requests from 256MB to 128MB on Wed
May 6th starting at 8 UTC, and from 128MB to 64MB on Tue May 12th starting
at 8 UTC. We will be restarting webservice tools as part of the deployment
and no action is required on tool maintainers' part.
The details of this work can be found at
https://phabricator.wikimedia.org/T420565
best,
Filippo
--
*Filippo Giunchedi*
Staff Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
In the last couple of weeks, we have replaced[0] the previous software
responsible for routing HTTP requests to individual Toolforge tools
(ingress-nginx[1], which is now end-of-life) with a new one
(Istio[2]). The primary visible change is that error pages in cases
where a tool has a configured web service that is not working for one
reason or another are now significantly more bare-bones, those might
be improved in the future.
[0]: https://phabricator.wikimedia.org/T392356
[1]: https://github.com/kubernetes/ingress-nginx/
[2]: https://istio.io/
If you are using the Kubernetes API directly[3] to configure tool
routing, you will need to migrate from old Ingress objects to new
Gateway API[4] -provided HTTPRoute[5] objects.
[3]: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes
[4]: https://gateway-api.sigs.k8s.io/
[5]: https://gateway-api.sigs.k8s.io/api-types/httproute/
taavi
--
Taavi Väänänen (he/they)
Site Reliability Engineer, Tools Infrastructure
Wikimedia Foundation
Summary:
The Cloud Services implementation of Superset
(https://superset.wmcloud.org) is currently unsupported by WMF staff. We
would welcome a volunteer stepping up to maintain it, but without a new
home the service will be shut down at the end of March.
If you would like to take on ownership of this service, please reach out
to WMCS staffor comment on the phabricator task[1]. The day-to-day
maintenance load is minimal, but someone will need to keep track up
updates, security concerns and a potential future of defense against AI
scraping.
Background:
There has always been demand from researchers and volunteers to run
one-off queries against Wikimedia datasets. Quarry.wmcloud.org was built
and launched by the WMF more than 10 years ago. Originally a one-person
skunkworks project, it quickly caught on in popularity as one of the
most trafficked services run by Wikimedia Cloud Services.
A few years ago, we launched superset.wmcloud.org[0] as an intended
Quarry replacement. We soon learned that despite being a less polished,
home-made tool, Quarry provides many features that Superset does not,
and most users resisted migration away from it. Superset has only a
handful of active users.
At this point, Superset is essentially unsupported by WMF staff. As part
of ongoing efforts to improve our support for Toolforge and other more
popular services we will be either shutting it off or transferring
ownership of Superset in a few weeks[1].
[0] https://phabricator.wikimedia.org/T169452
[1] https://phabricator.wikimedia.org/T416373
Some maintenance on Wiki Replicas [0] hosts is planned for next
Tuesday, April 21st, starting from 06:00 UTC and lasting up to 12
hours (hopefully it will take less!)
This will affect only section s6, which contains the following databases:
* frwiki
* jawiki
* labswiki
* ruwiki
For the duration of the maintenance, it will not be possible to query
those databases from Wiki Replicas, including from Toolforge, Quarry
or PAWS.
This is part of a larger refresh of Wiki Replicas hosts [1], that we
are carrying out trying to minimize as much as possible the impact to
end users. This time we are unfortunately required to take down one
section (s6) because while we have two copies of each section, one of
them had a hardware failure last week. [2]
Thanks for your understanding while we carry out this essential
maintenance work.
[0] https://wikitech.wikimedia.org/wiki/Help:Wiki_Replicas
[1] https://phabricator.wikimedia.org/T409557
[2] https://phabricator.wikimedia.org/T422813
Francesco
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Tools Platform
Wikimedia Foundation
tl;dr: new trove database versions are coming! Existing trove db servers
will continue to run as before. Please provide feedback about the specifics.
--
Sometime in the next few weeks I'd like to make a few updates to the
databases supported on cloud-vps. Unless you have your own private
database, toolforge users can ignore this email. If you ever visit the
'databases' tab in Horizon please read on!
Right now we support the following database engines:
mariadb10.5.10 (21 instances)
mysql5.7.29(4 instances)
postgresql 12.7 (13 instances)
I propose to deprecate all three of those engines, and replace them with:
mariadb12.2.2
postgres18
Here's how that would look, from a user standpoint:
1) **Any existing database instances will continue to run and operatein
Horizonas before.**
2) It will no longer be possible to create new database instances with
the deprecated engines.
3) It will no longer be possible to restore backups of databases using
the old engine without admin intervention -- if we do this, I will be on
hand to do manual restores if necessary. This restriction is bound up
with #2 above; either both rules stay or both rules go.[0]
4) There will be no automatic or in-place upgrade path provided from old
version to new version. Any upgrades, if needed, will have to be
performed 'by hand' with something like an outside-of-openstack dump and
restore. This pains me but I've spent some time testing and the internal
storage formats are too different for a smooth transition.
5) It will no longer be possible to create new mysql databases. Mysql is
not very popular, and as far as I know only provides a functional subset
of mariadb.
I'm making a lot of assumptions about you, the users, in this proposal,
so please speak up if (for example) you perform frequent backup
restores, or have a use case that requires mysql. You can respond to
this email, or follow up on the associated phab task[1].
If this email provokes only silence or agreement, then I will make these
changes on or near April 20.
[0] Currently only a total of 9 trove backups exist (and only 4 in the
last year), so I suspect no one is really relying on this service on the
regular.
[1] https://phabricator.wikimedia.org/T420737
(If you don't work with the image or oldimage tables, you can ignore
this message)
Hello,
in [1] we announced that finally we are implementing the changes
described in T28741[2].
This message provides an update on that announcement.
Currently, every current version of an image has a row in the image
table and if there are older versions of that file, those rows could be
found in the oldimage table. These two tables (image and oldimage) will
be dropped in around two months. The replacement will be two main
tables: file and filerevision. Every file will have a row in the file
table describing the name and the type. Every version of the file
(current and old) will have a row in filerevision describing the
file-specific information such as its size or the hash of the file,
similar to the existing distinction between pages and revisions. Another
improvement is that every file and file revision will get a unique auto
increment id simplifying many operations and queries.
The new tables are already accessible in wikireplicas and fully populated.
The old tables are currently also still accessible in wikireplicas.
However, if your tools or queries access image or oldimage directly,
please update them to use the new schema before 28 May. After that date,
the old tables will be removed from wikireplicas.
Thank you,
Alexander Vorwerk — IRC: Zabe
[1] -
https://lists.wikimedia.org/hyperkitty/list/cloud@lists.wikimedia.org/messa…
[2] - https://phabricator.wikimedia.org/T28741