This comes from a new database created called:
itwikivoyage_maintain
What is this? I cannot find anything related to that in phabricator.
Created https://phabricator.wikimedia.org/T422779 for tracking.
On Thu, Apr 9, 2026 at 7:11 AM root <root(a)clouddb1017.eqiad.wmnet> wrote:
> Private data detected at clouddb1017 for s3 check:
> /var/log/private_data_report_clouddb1017.log_s3
>
Hi,
We've found a bug that is being triggered in clouddb hosts with an insane
statement that queries 6 tables across 694 different wikis. Which makes the
server reach the stack limit and produces a segfault. This is being tracked
at: https://phabricator.wikimedia.org/T420177 (which I restricted it as a
security task for now).
I've been working with Marko and Dave from MariaDB and they've already
provided a patch which I compiled and packaged yesterday night (but didn't
install yet as we needed to collect more info on the crashes)
However, until we are fully sure it is fixed, I am going to downgrade to
10.11.13 which is the latest stable version where this didn't happen - this
will be tracked at https://phabricator.wikimedia.org/T421826
We believe the bug was introduced at:
https://jira.mariadb.org/browse/MDEV-35816
If you want to follow the bug I sent to MariaDB which includes stack
traces, the involved tabels and the insane query, please check:
https://jira.mariadb.org/browse/MDEV-39209
*This only affects clouddb web hosts.*
If you've got any questions, let me know!
Manuel.
The process for Cloud VPS quota requests used to state that "Requests
that represent an increase of more than double the quota or more than
300GB of storage should also be reviewed at the weekly meeting."
I don't think this was strictly followed in the past, and it could
block even relatively small requests for up to one week.
I boldly edited the Clinic Duties wiki page [0] to suggest those
requests should have two "+1"s instead. If you disagree with this
change, please let me know and we can discuss it during the next team
meeting.
[0] https://wikitech.wikimedia.org/w/index.php?title=Wikimedia_Cloud_Services_t…
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
Hi!
Just a heads up, I'm planning on upgrading toolsbeta k8s to 1.30[1] on monday
at around 08:00 UTC time.
I'll update live on irc during the upgrade, note that if you want to
deploy/test/run anything there it might stop working momentarily.
[1] https://phabricator.wikimedia.org/T402377
--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
There is a new Toolsbeta bastion running trixie, toolsbeta-bastion-7.
I've updated the login.toolsbeta.org service alias to point to it, the
SSH key fingerprint can be verified from [0].
[0]: https://toolsbeta-static.wmcloud.org/admin/fingerprints/
If you use some other name than `login.toolsbeta.org` to connect,
please update your config/memory/bash history. There's a variety of
old names under other domains that I plan to remove when removing the
old bastion instead of updating them.
Taavi
--
Taavi Väänänen (he/him)
Site Reliability Engineer, Cloud Services
Wikimedia Foundation
Apologies in advance for a possibly ranty email.
I'm again seeing some major Toolforge developments happening with major
decisions being made in private. A few examples:
* Infrastructure IaC/OpenTofu migration (with an intern assigned to it)
popping out of nowhere to implementation tasks
https://phabricator.wikimedia.org/T390056 with no public/formal
discussion whether we want to replace the current tooling with that.
* UI project starting an "investigation" task
(https://phabricator.wikimedia.org/T383146) with a clear bias towards a
specific solution, based on unspecific "previous discussions", which
were then hidden from the description when I asked for documents
relating those to be made public.
* The Toolforge CLI consolidation project apparently has switched
implementation languages compared to the last decision request,
apparently based on "a team decision" that was apparently made "a few
months" ago but never communicated anywhere as far as I can tell:
https://phabricator.wikimedia.org/T356262#10722410
First, could we please again stop doing major decisions in private?
Second, is there something in our processes or workflows that we could
improve so that things wouldn't always immediately start going more
hidden and private every time I stop paying close attention to this?
Taavi
Hi there,
the tofu-infra project gained support for managing quotas, see
https://phabricator.wikimedia.org/T371391
Openstack quotas can now be established in several ways:
* traditionally via the wmcs-openstack CLI
* using the cookbook to manage quotas
* using tofu-infra
If a quota is set via tofu-infra, it will override all the others, as far as I
understand how they work.
If a quota is not set via tofu-infra, then whatever quota was already
established should be in action.
I have not back-filled, nor have plans to do so, previous quotas into tofu-infra.
regards.
Hello,
there will be a network maintenance operation happening on Thursday 2025-03-27
12:30 UTC that may impact all Wikimedia Cloud Services, including:
* Cloud VPS
* Toolforge
* PAWS
* and any others
The planned operation window will last for 1h, from 12:30 UTC to 13:30 UTC.
We have prepared the required changes and we are expecting no outages. But the
operation is sensitive and we would like you to be informed of potential service
disruptions as a result of the operations.
If you want more technical details, we are using a Phabricator ticket:
https://phabricator.wikimedia.org/T389958
regards.
Hello,
If you have a Cloud VPS project and use opentofu to manage virtual resources,
this message is for your -- otherwise feel free to ignore. In particular,
Toolforge users are free to ignore this message.
The main Cloud VPS virtual network resource name is changing:
* from: lan-flat-cloudinstances2b
* to: VLAN/legacy
If your opentofu / terraform code mentions this network name, you will need to
update it.
See also Phabricator ticket: https://phabricator.wikimedia.org/T389942
regards.