If you use Wiki Replicas to query wikidata termstore tables (wbt_*),
be aware that those tables are not currently getting live updates.
The Data Persistence team is working on setting up the new "x3"
section in Wiki Replicas hosts,[0] that is expected to take multiple
days to complete.
In the meantime, you can start using the following new endpoints:
- termstore.wikidatawiki.analytics.db.svc.wikimedia.cloud
- termstore.wikidatawiki.web.db.svc.wikimedia.cloud
Those endpoints are already returning data, but the data returned will
not be up-to-date. As soon as the replication of those tables is
restarted on Wiki Replicas, those endpoints will start returning live
data.
For more information, refer to the News page on Wikitech.[1]
[0] https://phabricator.wikimedia.org/T390954
[1] https://wikitech.wikimedia.org/wiki/News/2025_Wikidata_term_store_database_…
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
Starting next month, we are going to upgrade all the Wiki Replicas [0]
servers from MariaDB 10.6 to MariaDB 10.11.
We don't expect any impact to users, as there are no big breaking
changes in the new version. If you are curious, you can check the full
list of changes on the MariaDB website. [1]
I will send an update before we start the upgrade, you can also follow
the upgrade on Phabricator. [2]
[0] https://wikitech.wikimedia.org/wiki/Help:Wiki_Replicas
[1] https://mariadb.com/kb/en/changes-improvements-in-mariadb-1011/
[2] https://phabricator.wikimedia.org/T394372
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
I will be upgrading the cloud-vps openstack install on Thursday,
beginning around 16:00 UTC. Here's what to expect:
- Intermittent Horizon and API downtime (maybe an hour or two total)
- Inability to schedule new VMs (also for an hour or two)
Toolforge users will be unaffected by this outage. Existing, running
services and VMs on cloud-vps should also be unaffected.
This is the upgrade to version 2025.1 'epoxy', tracked at
https://phabricator.wikimedia.org/T390914
-Andrew + the WMCS team
tl;dr: a random assortment of security group/firewall rules were briefly
broken which caused networking issues for some cloud-vps services. This
is now resolved.
This outage began at 12:51 PMUTC, was mostly resolved at 13:25 UTC and
full resolved at 13:35UTC.
Full (still short) story:
As part of ongoing efforts to make all of cloud-vps work with IPv6, we
just now ran an automated script to expand existing securitygroup rules
to ipv6 access. A bug in that script effectively destroyed existing
rules rather than updating them, which caused many open doors to
unceremoniously slam shut.
In order to recover from this as quickly as possible, we restored a
backup of the network database from earlier today (04:28 UTC). This
restored service but also wiped out any networking changes that might
have been made during that time. This means that if you happen to have
changed your security groups since then, you will need to repeat your
changes.
The still-in-progress incident report can be found here:
https://wikitech.wikimedia.org/wiki/Incidents/2025-05-07_cloud-vps_security…
- Andrew +WMCS staff, Slightly embarassed
tl;dr: a random assortment of security group/firewall rules were briefly
broken which caused networking issues for some cloud-vps services. This
is now resolved.
This outage began at 12:51 PMUTC, was mostly resolved at 13:25 UTC and
full resolved at 13:35UTC.
Full (still short) story:
As part of ongoing efforts to make all of cloud-vps work with IPv6, we
just now ran an automated script to expand existing securitygroup rules
to ipv6 access. A bug in that script effectively destroyed existing
rules rather than updating them, which caused many open doors to
unceremoniously slam shut.
In order to recover from this as quickly as possible, we restored a
backup of the network database from earlier today (04:28 UTC). This
restored service but also wiped out any networking changes that might
have been made during that time. This means that if you happen to have
changed your security groups since then, you will need to repeat your
changes.
The still-in-progress incident report can be found here:
https://wikitech.wikimedia.org/wiki/Incidents/2025-05-07_cloud-vps_security…
- Andrew +WMCS staff, Slightly embarassed