Next Tuesday we'll be replacing[0] the cloud gateway nodes that serve
the entire cloud-vps and toolforge network. If all goes well this will
only cause a quick connection reset but we're scheduling a window of
potential downtime in case things do not go well.
This work is in response to hardware errors that have been appearing on
one of the existing gateway hosts[1].
-Andrew + the WMCS team
[0] https://phabricator.wikimedia.org/T382356
[1] https://phabricator.wikimedia.org/T382220
Due to a latent configuration error, many VMs need to be rebooted. The
affected VMs are working fine now, but if restarted they will start with
an incorrect block storage config and be unable to launch, requiring
manual repair.
I am planning to migrate + reboot all the VMs listed next Tuesday, which
will rebuild and repair the broken config. If you find one of these VMs
in an inconsistent state in the meantime, please contact a WMCS root and
refer them to the related task[0] which explains the somewhat tedious
process of revival.
Some of the VMs to be rebooted are nfs servers, which may cause service
interruptions for projects relying on those servers. Specifically:
1b001a01-0a37-4d81-9dd6-1339b6391f37 paws-nfs-1.paws
c18ab12b-fa18-4a29-a074-40535930f6b0 k8s-nfs.quarry
a1cb92ab-6083-4465-81a9-a283918f13eb toolsbeta-nfs-3.toolsbeta
cf6dc802-f95c-48c1-a07b-8791185a5e45 fastcci-nfs-1.fastcci
The rest of the VMs that need reboots are:
4b932658-2b4d-4157-b62e-1e73faaed8aa quarry-db-02.trove
14be24bb-a36b-4627-9254-d02a0987db83 data.wikipathways
14be24bb-a36b-4627-9254-d02a0987db83 data.wikipathways
7dea739e-7f9a-48ed-8169-e88300c77092 gitlab-runner-1003.devtools
9aa7e1f6-1e53-4c54-814a-1c6e7b8e55bb dannyb.wildcat
07543105-1336-4f7f-ae04-5c7989a0bfb5 larynx-be-01.text-to-speech
1bff1756-ce9a-40b6-8783-a01910dff1b6 storage.qrank
b26f51d1-b1e6-46a6-877d-3d43acc1ccac wikiapiary.wikiapiary
3c53fc19-f19f-4c6a-8b23-ab8edb149994 libup-db02.trove
7b1a49d7-2df1-4011-942d-80da14e38f7d buttercup.wikifunctions
4222d13d-aea4-4168-8542-03b5e91f2af8 tools-imagebuilder-2.tools
23e4ccb3-6146-482d-ad4e-1e83e78d1482 deployment-snapshot05.deployment-prep
c3be1a8e-94f4-4db8-b3d2-99f74f034ea5 loggerdiscordbot.trove
94f64547-3537-4eae-b7c8-f33f0d5e58bd
cloudinfra-internal-puppetserver-1.cloudinfra
32e12f3c-bc0c-4dc1-8ada-abd6e03d1f75 tools-puppetserver-01.tools
d277044f-a5db-4939-b1e4-5a142116f9d0 quarry-k8s.trove
c4f71bdf-4d88-4f2c-a0bc-d358975a91eb free3.dwl
0126dc3f-02c9-4b07-8550-de103724fdc3 ifis.trove
ba0e8a43-352b-48cd-bc86-8f2374fd7447 logging-opensearch-hdd-01.logging
8071f964-80be-41b0-b042-14024965f1ba tools-docker-registry-7.tools
c4e76fe0-82c5-4188-8e35-0f63b6d4a216 logging-opensearch-ssd-02.logging
ca6525d1-1863-4ab0-b0ef-07c35d063324 taxonbot3.dwl
0038b497-9331-4493-a688-5c85830494e9 tools-docker-registry-8.tools
27c89a15-ee59-4cb2-81f7-845d582ccc88 mwv-puppetserver-01.mediawiki-vagrant
a8b2ff57-9841-434b-b233-d8c0064ddb1b mailman-puppetserver-1.mailman
fb7fba39-5cef-44ea-a6f3-5b7145f446e8 legacy.wikiapiary
fb7fba39-5cef-44ea-a6f3-5b7145f446e8 legacy.wikiapiary
b93776fd-b69b-459e-9960-ee69a6b52732
cloudinfra-cloudvps-puppetserver-1.cloudinfra
337b88c3-9009-402d-a7d5-182b1b027fbb copypatrol-dev-db-01.trove
eb811cae-2a12-43eb-9d1f-33ee250fe51a copypatrol-prod-db-01.trove
c738d3bb-8bd4-4807-8651-78698923385e thistle.trove
c02583d4-fa94-4a5c-a4a2-1a50872d45a5 mariadb.trove
db1e3259-163c-4a3f-9b5b-a8a37bbb6f5d db01.trove
037795b1-0d21-41e8-840e-ac379426d1e9 osmit-tm4.osmit
eb6f591f-69b2-4f16-a7eb-3740b5fd074f db.933ad3ff1e264aada56e6bc3ed9e08f3
50ea5f92-8cb9-4e21-bc65-6fe0b6dd9550
superset-127-jxhvhh7bzlrl-master-0.superset
2faa7a26-c56a-4219-b658-b680212cb64c
superset-127-jxhvhh7bzlrl-node-0.superset
70a287ed-3071-4df9-ba1a-ac512cf8749a
superset-127-jxhvhh7bzlrl-node-1.superset
d182fffb-9f32-41de-96cf-ce5cc5fc95c2 citefix-db.trove
fff879b9-8300-4a11-9cf6-3d424be9ffa3 cloudinfra-db04.cloudinfra
47789d5d-4777-455f-8e00-4d24507fe5e7 worker-1.spacemedia
e9b6fd4c-d5a3-40dc-968a-ddfcf2222389 worker-2.spacemedia
f50f9084-d034-4dae-84fb-80980868335a tools-elastic-4.tools
16ce1807-dd8e-4b87-b4ba-558d2719bbe5 tools-elastic-6.tools
39737a21-257d-4267-9aa6-d2c411501b5f peony-database.globaleducation
c8ccdc13-f8b0-4e37-9a2b-795f0bff9f59 videocuttool-bookworm-new.videocuttool
c8ccdc13-f8b0-4e37-9a2b-795f0bff9f59 videocuttool-bookworm-new.videocuttool
bfad7fbd-53db-4604-aa38-19ffa3e3da02 harbordb.trove
93742dcf-1d18-4c28-8ba0-24ea3219f144 wmcz-stats-test03.wmcz-stats
17c5890b-d783-4f3c-938d-0a3ff2e2e3bc wmcz-stats-wikinside01.wmcz-stats
f37f17cf-ee65-4a40-888e-3500120d566d ml-testing.machine-learning
7fd6ada6-2d78-443b-84ce-57aec7cd7a4a ia-upload-prod2.wikisource
f75a5008-1b83-4bc2-8cb7-c74f5262d053 pcc-db1002.puppet-diffs
d1818008-69c9-4a7b-b642-baeb313df663 patchdemo4-production.catalyst
c9bff3a7-f77b-4fea-a31c-b4fb77c4e9e2 deployment-jobrunner05.deployment-prep
de63de80-8f02-47e0-9ad2-cfe407ca99be deployment-mediawiki14.deployment-prep
5ff144a5-bb68-44b7-9d0f-39df6b9d3063 deployment-mediawiki13.deployment-prep
93c9a448-d01c-4229-8cf3-29b8a0b765cb deployment-parsoid14.deployment-prep
fe1ff5d8-813c-4d06-ae3a-46c05715d305 tools-services-06.tools
c99e6edf-89ab-4300-b637-80e11e55438e tools-harbordb.trove
9b5de945-4495-4516-8672-13e525acc557 encoding01.video
f9bee5ff-85f5-4cdf-876a-3f4e2e3f3c8e encoding03.video
5a7dcadf-c88e-4d4b-9e44-42e3d636c4ba encoding05.video
85d30118-6397-4bb1-a652-31bb48a1fb2b traffic-puppetserver-bookworm.traffic
5ed094b0-5a2f-42d3-9fe3-a93237351475 encoding06.video
fcd555ae-c19c-4b2e-8785-8a9447b263d8 humaniki-prod-bw.wikidumpparse
de830403-c93b-4340-9467-d1d8f7ea1628 sample-complex-app-db.trove
3a4d671b-5aef-49ef-b6b4-5cb0c7d00e38 toolsbeta-harbor-2.toolsbeta
421dd73e-8c34-4962-a4e2-61c8997b27d1
quarry-127a-g4ndvpkr5sro-master-0.quarry
a1a12a26-55cd-41cd-9696-74773b4db7f0
deployment-wikikube-v127-zwallxbnux67-master-0.deployment-prep
8b99ac12-660d-4267-b841-2d458f8dd9a4
deployment-wikikube-v127-zwallxbnux67-node-0.deployment-prep
65c14b90-8b59-498a-851d-b3c4259fb134 paws-127a-m3mctzr7itba-master-0.paws
3599d6d5-89c2-40ba-95dc-c48c066bd56b paws-127a-m3mctzr7itba-node-2.paws
15f8fdeb-46f9-4853-bdd8-e8c074dce507 paws-127a-m3mctzr7itba-node-3.paws
ecac332c-d805-482c-ba71-93838c73b100 paws-127a-m3mctzr7itba-node-1.paws
8373fb59-3b0a-4259-868b-1b4b81c3d10c paws-127a-m3mctzr7itba-node-4.paws
caf6628a-f9f7-4f1b-a42a-3cc79b12a758 paws-127a-m3mctzr7itba-node-0.paws
caf6628a-f9f7-4f1b-a42a-3cc79b12a758 paws-127a-m3mctzr7itba-node-0.paws
ecac332c-d805-482c-ba71-93838c73b100 paws-127a-m3mctzr7itba-node-1.paws
[0] https://phabricator.wikimedia.org/T383583
I'm cross-posting what was included in today's "Tech News", for people
who are not subscribed to that list:
"Toolforge tool maintainers can now share their feedback on Toolforge
UI, an initiative to provide a web platform that allows creating and
managing Toolforge tools through a graphic interface, in addition to
existing command-line workflows. This project aims to streamline
active maintainers’ tasks, as well as make registration and deployment
processes more accessible for new tool creators. The initiative is
still at a very early stage, and the Cloud Services team is in the
process of collecting feedback from the Toolforge community to help
shape the solution to their needs."
Read more and share your thoughts about Toolforge UI at this link:
https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme…
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
Summary:
Many toolforge jobs just now failed and/or were interrupted. Everything
is now either back online or coming rapidly back online.
Explanation:
In response to a completely unrelated cloud-vps infra issue
(https://phabricator.wikimedia.org/T383583) I needed to migrate many VMs
to new hypervisors. I used an updated openstack command for the
migration, which turned out to unexpectedly reboot the migrated VMs as
part of the migration.
Among those VMs rebooted was the toolforge NFS server. This caused the
inevitable storm of disconnections, failed mounts, and filesystem
lock-ups. This outage was prolonged by the fact that the reboots /also/
temporarily broke puppet which would have otherwise immediately brought
the NFS server back online.
After staged restarts of various services everything began to recover.
As usual, we are now rebooting tools nfs workers in order to reset any
locked up nfs file handles.
Sorry for the outage! Please follow up here or online if you see any
continuing issues. The outage itself is documented as
https://phabricator.wikimedia.org/T383625
Tl;dr:
Please let me know if you encounter changes in network connectivity
between cloud-vps hosts, specifically when routing to floating IPs.
Complicated explanation:
A previous version of our network setup prevented routing between
private and floating IPs within cloud-vps. To work around this we run an
agent called 'labs-ip-aliaser' which insertes corresponding private IPs
for any dns lookup initiating from within cloud-vps that would otherwise
return a floating IP.
So, for instance, if you did something like
$ ping hostname.wmcloud.org
you would actually be pinging vmname.project.eqiad.wikimedia.cloud
instead, because that worked and pinging the wmcloud.org address did not.
We have since refactored the network so that now reaching public IPs
works as expected. So, later in the week I'm going to remove the
labs-ip-aliaser hack so that floating-ip-associated hostnames can
resolve to the actual floating IPs as expected. This change should be
almost entirely invisible to cloud-vps services, but please notify me if
you find an interesting edge-case!
https://phabricator.wikimedia.org/T374129https://gerrit.wikimedia.org/r/c/operations/puppet/+/1105443/7