I'm running reboots on most of these VMs now. I've (tediously) hand-tuned the nfs servers in order to avoid the reboot.
On 1/15/25 10:28 AM, Andrew Bogott wrote:
Correction: this is happening Tuesday, 2025-01-21.
A helpful reader pointed out that as much as I might like to, I cannot reset these VMs in the past but only in the future :)
On 1/14/25 3:24 PM, Andrew Bogott wrote:
Due to a latent configuration error, many VMs need to be rebooted. The affected VMs are working fine now, but if restarted they will start with an incorrect block storage config and be unable to launch, requiring manual repair.
I am planning to migrate + reboot all the VMs listed next Tuesday, which will rebuild and repair the broken config. If you find one of these VMs in an inconsistent state in the meantime, please contact a WMCS root and refer them to the related task[0] which explains the somewhat tedious process of revival.
Some of the VMs to be rebooted are nfs servers, which may cause service interruptions for projects relying on those servers. Specifically:
1b001a01-0a37-4d81-9dd6-1339b6391f37 paws-nfs-1.paws c18ab12b-fa18-4a29-a074-40535930f6b0 k8s-nfs.quarry a1cb92ab-6083-4465-81a9-a283918f13eb toolsbeta-nfs-3.toolsbeta cf6dc802-f95c-48c1-a07b-8791185a5e45 fastcci-nfs-1.fastcci
The rest of the VMs that need reboots are:
4b932658-2b4d-4157-b62e-1e73faaed8aa quarry-db-02.trove 14be24bb-a36b-4627-9254-d02a0987db83 data.wikipathways 14be24bb-a36b-4627-9254-d02a0987db83 data.wikipathways 7dea739e-7f9a-48ed-8169-e88300c77092 gitlab-runner-1003.devtools 9aa7e1f6-1e53-4c54-814a-1c6e7b8e55bb dannyb.wildcat 07543105-1336-4f7f-ae04-5c7989a0bfb5 larynx-be-01.text-to-speech 1bff1756-ce9a-40b6-8783-a01910dff1b6 storage.qrank b26f51d1-b1e6-46a6-877d-3d43acc1ccac wikiapiary.wikiapiary 3c53fc19-f19f-4c6a-8b23-ab8edb149994 libup-db02.trove 7b1a49d7-2df1-4011-942d-80da14e38f7d buttercup.wikifunctions 4222d13d-aea4-4168-8542-03b5e91f2af8 tools-imagebuilder-2.tools 23e4ccb3-6146-482d-ad4e-1e83e78d1482 deployment-snapshot05.deployment-prep c3be1a8e-94f4-4db8-b3d2-99f74f034ea5 loggerdiscordbot.trove 94f64547-3537-4eae-b7c8-f33f0d5e58bd cloudinfra-internal-puppetserver-1.cloudinfra 32e12f3c-bc0c-4dc1-8ada-abd6e03d1f75 tools-puppetserver-01.tools d277044f-a5db-4939-b1e4-5a142116f9d0 quarry-k8s.trove c4f71bdf-4d88-4f2c-a0bc-d358975a91eb free3.dwl 0126dc3f-02c9-4b07-8550-de103724fdc3 ifis.trove ba0e8a43-352b-48cd-bc86-8f2374fd7447 logging-opensearch-hdd-01.logging 8071f964-80be-41b0-b042-14024965f1ba tools-docker-registry-7.tools c4e76fe0-82c5-4188-8e35-0f63b6d4a216 logging-opensearch-ssd-02.logging ca6525d1-1863-4ab0-b0ef-07c35d063324 taxonbot3.dwl 0038b497-9331-4493-a688-5c85830494e9 tools-docker-registry-8.tools 27c89a15-ee59-4cb2-81f7-845d582ccc88 mwv-puppetserver-01.mediawiki-vagrant a8b2ff57-9841-434b-b233-d8c0064ddb1b mailman-puppetserver-1.mailman fb7fba39-5cef-44ea-a6f3-5b7145f446e8 legacy.wikiapiary fb7fba39-5cef-44ea-a6f3-5b7145f446e8 legacy.wikiapiary b93776fd-b69b-459e-9960-ee69a6b52732 cloudinfra-cloudvps-puppetserver-1.cloudinfra 337b88c3-9009-402d-a7d5-182b1b027fbb copypatrol-dev-db-01.trove eb811cae-2a12-43eb-9d1f-33ee250fe51a copypatrol-prod-db-01.trove c738d3bb-8bd4-4807-8651-78698923385e thistle.trove c02583d4-fa94-4a5c-a4a2-1a50872d45a5 mariadb.trove db1e3259-163c-4a3f-9b5b-a8a37bbb6f5d db01.trove 037795b1-0d21-41e8-840e-ac379426d1e9 osmit-tm4.osmit eb6f591f-69b2-4f16-a7eb-3740b5fd074f db.933ad3ff1e264aada56e6bc3ed9e08f3 50ea5f92-8cb9-4e21-bc65-6fe0b6dd9550 superset-127-jxhvhh7bzlrl-master-0.superset 2faa7a26-c56a-4219-b658-b680212cb64c superset-127-jxhvhh7bzlrl-node-0.superset 70a287ed-3071-4df9-ba1a-ac512cf8749a superset-127-jxhvhh7bzlrl-node-1.superset d182fffb-9f32-41de-96cf-ce5cc5fc95c2 citefix-db.trove fff879b9-8300-4a11-9cf6-3d424be9ffa3 cloudinfra-db04.cloudinfra 47789d5d-4777-455f-8e00-4d24507fe5e7 worker-1.spacemedia e9b6fd4c-d5a3-40dc-968a-ddfcf2222389 worker-2.spacemedia f50f9084-d034-4dae-84fb-80980868335a tools-elastic-4.tools 16ce1807-dd8e-4b87-b4ba-558d2719bbe5 tools-elastic-6.tools 39737a21-257d-4267-9aa6-d2c411501b5f peony-database.globaleducation c8ccdc13-f8b0-4e37-9a2b-795f0bff9f59 videocuttool-bookworm-new.videocuttool c8ccdc13-f8b0-4e37-9a2b-795f0bff9f59 videocuttool-bookworm-new.videocuttool bfad7fbd-53db-4604-aa38-19ffa3e3da02 harbordb.trove 93742dcf-1d18-4c28-8ba0-24ea3219f144 wmcz-stats-test03.wmcz-stats 17c5890b-d783-4f3c-938d-0a3ff2e2e3bc wmcz-stats-wikinside01.wmcz-stats f37f17cf-ee65-4a40-888e-3500120d566d ml-testing.machine-learning 7fd6ada6-2d78-443b-84ce-57aec7cd7a4a ia-upload-prod2.wikisource f75a5008-1b83-4bc2-8cb7-c74f5262d053 pcc-db1002.puppet-diffs d1818008-69c9-4a7b-b642-baeb313df663 patchdemo4-production.catalyst c9bff3a7-f77b-4fea-a31c-b4fb77c4e9e2 deployment-jobrunner05.deployment-prep de63de80-8f02-47e0-9ad2-cfe407ca99be deployment-mediawiki14.deployment-prep 5ff144a5-bb68-44b7-9d0f-39df6b9d3063 deployment-mediawiki13.deployment-prep 93c9a448-d01c-4229-8cf3-29b8a0b765cb deployment-parsoid14.deployment-prep fe1ff5d8-813c-4d06-ae3a-46c05715d305 tools-services-06.tools c99e6edf-89ab-4300-b637-80e11e55438e tools-harbordb.trove 9b5de945-4495-4516-8672-13e525acc557 encoding01.video f9bee5ff-85f5-4cdf-876a-3f4e2e3f3c8e encoding03.video 5a7dcadf-c88e-4d4b-9e44-42e3d636c4ba encoding05.video
85d30118-6397-4bb1-a652-31bb48a1fb2b traffic-puppetserver-bookworm.traffic 5ed094b0-5a2f-42d3-9fe3-a93237351475 encoding06.video fcd555ae-c19c-4b2e-8785-8a9447b263d8 humaniki-prod-bw.wikidumpparse de830403-c93b-4340-9467-d1d8f7ea1628 sample-complex-app-db.trove 3a4d671b-5aef-49ef-b6b4-5cb0c7d00e38 toolsbeta-harbor-2.toolsbeta 421dd73e-8c34-4962-a4e2-61c8997b27d1 quarry-127a-g4ndvpkr5sro-master-0.quarry a1a12a26-55cd-41cd-9696-74773b4db7f0 deployment-wikikube-v127-zwallxbnux67-master-0.deployment-prep 8b99ac12-660d-4267-b841-2d458f8dd9a4 deployment-wikikube-v127-zwallxbnux67-node-0.deployment-prep 65c14b90-8b59-498a-851d-b3c4259fb134 paws-127a-m3mctzr7itba-master-0.paws 3599d6d5-89c2-40ba-95dc-c48c066bd56b paws-127a-m3mctzr7itba-node-2.paws 15f8fdeb-46f9-4853-bdd8-e8c074dce507 paws-127a-m3mctzr7itba-node-3.paws ecac332c-d805-482c-ba71-93838c73b100 paws-127a-m3mctzr7itba-node-1.paws 8373fb59-3b0a-4259-868b-1b4b81c3d10c paws-127a-m3mctzr7itba-node-4.paws caf6628a-f9f7-4f1b-a42a-3cc79b12a758 paws-127a-m3mctzr7itba-node-0.paws caf6628a-f9f7-4f1b-a42a-3cc79b12a758 paws-127a-m3mctzr7itba-node-0.paws ecac332c-d805-482c-ba71-93838c73b100 paws-127a-m3mctzr7itba-node-1.paws
_______________________________________________ Cloud-announce mailing list -- cloud-announce@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.o...