January 2025 - Cloud - lists.wikimedia.org

Several database changes happening in the coming months
by Amir Sarabadani 24 Apr '25

24 Apr '25

Hello, in the next coming months, these changes will happen in databases and the infrastructure. And it might affect you if you rely on them in your tools or queries. This list is ordered based on how soon the change will happen. We understand that updating your tools and systems can be time consuming, hence we are giving an advanced notice. I truly apologize for the inconvenience but many of these changes are needed to keep the site running smoothly. Image table redesign Around fourteen years after the creation of T28741 <https://phabricator.wikimedia.org/T28741>, we are implementing the changes described therein. Currently, every current version of an image has a row in the image table and if there are older versions of that file, those rows could be found in the oldimage table. These two tables (image and oldimage) will be dropped in around two months. The replacement will be two main tables: file and filerevision. Every file will have a row in the file table describing the name and the type. Every version of the file (current and old) will have a row in filerevision describing the file-specific information such as its size or the hash of the file, similar to the existing distinction between pages and revisions. Another improvement is that every file and file revision will get a unique auto increment id simplifying many operations and queries. You can check T28741 <https://phabricator.wikimedia.org/T28741> for more information. The new tables are already accessible in wikireplicas but the data hasn’t been fully migrated yet. Term store split out of wikidata’s database Wikidata’s database has been growing too fast and we need to move the term store (tables starting with wbt_) to a dedicated cluster to allow growth and improve wikidata’s performance by utilizing cache locality. The new section will be called x3 and you will be able to access it in wikireplicas but this also means you won’t be able to join these tables with the rest of wikidata’s database (such as page table) since they will be residing in two physically separate servers that also means most of your queries to wikidata’s database (and term store) will become faster. We are aiming for the switch to happen in three months’ time. You can follow the work in T351820 <https://phabricator.wikimedia.org/T351820>. Additionally, wb_type table will be dropped and the mapping will be hard-coded in the code instead. See gerrit:1110810 <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110810> for more details. This helped us simplify a lot of Wikibase code (example <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1110720>). Categorylinks normalization Categorylinks is the next table in the series of links tables being normalized via the linktarget table (parent ticket <https://phabricator.wikimedia.org/T300222>, RFC <https://phabricator.wikimedia.org/T222224>). Similar to templatelinks and pagelinks tables, cl_to will be dropped and instead the new field cl_target_id will point to lt_id in the linktarget table. We will also drop the cl_collation field and replace it with cl_collation_id which will point to the collation_id field on the new table we are introducing called collation. We are aiming to get this fully done by the end of the next quarter (end of June 2025) but it depends on how fast the migration script can operate and that’s outside of our control. You can follow the work in T299951 <https://phabricator.wikimedia.org/T299951>.It’s worth noting that after this migration is done, we will start working on the imagelinks table. Thank you -- *Amir Sarabadani (he/him)* Staff Database Architect Wikimedia Foundation <https://wikimediafoundation.org/>

1 1

[Cloud-announce] Project admins, please hard reboot your VMs (or I will)
by Andrew Bogott 06 Feb '25

06 Feb '25

The issue that resulted in partial VM reboots last week[0] (see "VM reboots coming Tuesday, 2024-01-20") turns out to be more widespread, affecting virtually all instances. The primary symptom is that it restricts our ability to drain and maintain WMCS hardware. In order to resolve the issue, all that's needed is a hard reboot of each VM. Note that a simple in-place reboot (for instance, issued at in the shell of the VM) does NOT resolve the issue. Hard reboots must be performed via Horizon, by selecting 'Hard Reboot Instance' on the Instances panel. So, please, at your convenience, log into Horizon and hard reboot any of your VMs that appear on the list below. For informal coordination I have also populated an etherpad[1] with the list of affected VMs. Anything that remains in need of a reboot by late next week (Thursday, February 6th) I will reboot for you. So, if you don't care when/if your VM is rebooted you can ignore this message :) Thank you! And, sorry for the inconvenience. We have a plan in place[2] which should prevent this issue from re-appearing in the future. -Andrew [0] https://phabricator.wikimedia.org/T383583 [1] https://etherpad.wikimedia.org/p/rmrebootsforcephmons [2] https://phabricator.wikimedia.org/T385288 -- [] account-creation-assistance: accounts-appserver6 9d7288ae-ad02-40a6-8976-495d63c040a0 [] account-creation-assistance: accounts-db7 8fcade67-7226-469a-8fac-dda6a08f24d0 [] account-creation-assistance: accounts-oauth-g 3d107f3e-1200-416f-9190-94b0cd9541b0 [] adiutor: testbed ed652064-c031-495b-bb4b-6ace51e2b01a [] adiutor: testbed-vagrant 671aae8e-3a57-47f6-bb44-9275b7b4516b [] analytics: gitlab-docker-runner 2122f5ae-3354-41fa-96e2-eed9efb25a4d [] analytics: growthbook 8e26bc28-39f0-49f8-8cc4-bd6b61411bd3 [] analytics: kdc 32a69b77-1dfc-4e30-a522-883abcaf43f0 [] appservers: j-tst-puppet-01 0c315541-2483-42bf-84f0-5ab8ee592359 [] auditlogging: syslog-client03 c0645c2a-e8d9-48cd-898b-4c466f1be8dc [] auditlogging: syslog-client04 2d6ab6a0-cb97-4ee7-bd8f-e7e8408c5db3 [] auditlogging: syslog-client05 d8ccc10d-3c77-4b30-8179-7498ac8cc17a [] auditlogging: syslog-server03 07e9058c-62c6-4a81-97a1-f36b11603689 [] auditlogging: syslog-server05 8e75ce49-694f-4c68-9d1a-10115e7f791b [] automation-framework: diffscan02 dba77a95-d482-43af-853a-7b72b309dd9e [] bastion: bastion-eqiad1-03 15700b39-5773-4fec-9349-47f16bc457bd [] bastion: bastion-eqiad1-04 20d097f7-3866-4f06-b3d6-cd7d37eb33b5 [] bastion: bastion-restricted-eqiad1-3 47b0da1d-e50a-42c1-8cd9-dc255cc2f1a3 [] bub2: bub2-instance 6cc04f18-9f59-41bb-9bcd-9aaa8bcc2d96 [] capacity-exchange: production d747c697-0895-43a0-93ee-2b5db340bb22 [] catalyst-dev: k3s-node01 6799679b-ac3f-4e07-8bd4-8c23640a5886 [] catalyst-qte: k3s-main 1ad1581e-24e0-4903-bda0-ef74fe5e1b61 [] catalyst-qte: k3s-node01 886df770-6c2a-4f38-8045-34fa5e695a69 [] citefix: citefix 709122ba-c8b5-48ba-9694-fbc89b1cf5f7 [] civicrm-prototype: puppetserver-01 d1a2fc36-1a82-46c1-bd01-d1382e54feee [] cloudinfra: enc-1 e94378df-7a48-4b11-b44a-bf69aaf132bd [] cloudinfra: etcd-discovery-1 926dc59c-3e85-4ae0-a65f-0140fb202a2a [] cloudinfra: mx-out05 a9c57526-aaed-4b93-9cbe-e992c6fc25e5 [] cloudinfra: ntp-03 dd448d31-6a7c-4202-afeb-88ba838c2d85 [] cloudinfra: ntp-04 7773ba13-e882-4780-ae0d-da8a39f26a3a [] cloudinfra: syslog-server-audit01 be0eb3db-cd9a-482a-a31b-40c0babf66ae [] codesearch: codesearch9 683ef4dd-491d-401b-b60e-88e2945e54ea [] collection-alt-renderer: mediawiki2latex 5001eac3-74c2-49a6-b081-195f68464c25 [] commtech: commtech-bot02 a2466526-104b-4f96-ada1-66d28aff7572 [] commtech: commtech-nsfw03 8db5b1a2-506a-4c90-924c-030005488308 [] commtech: pageviews01 0e4a8b46-a264-457b-8711-0bee2345e871 [] copypatrol: copypatrol-backend-prod-01 50d3e893-aee7-4591-9f2b-630107c6a036 [] copypatrol: dev01 df58d6c5-6dae-4b56-8507-dc2e00679834 [] copypatrol: prod01 a870b45e-e184-4833-9117-d49de7263d69 [] cvn: cvn-apache10 f5c20388-c623-4ad4-914c-db565654bf9f [] cvn: cvn-app10 1c7097fe-f721-46ef-928f-2d975d96afb6 [] cyberbot: cyberbot-exec-iabot-01 58bff802-41bc-4ae6-b832-30636d25e0e8 [] cyberbot: cyberbot-exec-iabot-02 eb1784d4-f013-43d6-97d4-2a4ae5f92af3 [] cyberbot: cyberbot-web-iabot-01 af3129d7-3d4a-43b9-8156-53eb636f27a5 [] dashiki: dashiki-03 595b3e7d-13cc-4695-9f60-a86618541cc8 [] dashiki: dashiki-staging-03 525bc74f-950c-427b-90b8-1db821f8d68b [] deployment-prep: deployment-acme-chief05 9bc3b4dd-bea1-47c4-8f54-d4f9c918c19f [] deployment-prep: deployment-acme-chief06 89296f92-ecfe-4fdd-bec9-10ff30b27f7d [] deployment-prep: deployment-bastion bdde8967-2caa-4dd1-ae27-829e808eb64a [] deployment-prep: deployment-cache-text08 7102217c-5d7f-4758-8936-fd72b578f3e8 [] deployment-prep: deployment-cache-upload08 11bc7c03-20a9-4488-86c7-ce4b7faa9283 [] deployment-prep: deployment-changeprop-1 0cedcc02-56cd-42aa-9017-add8a4291793 [] deployment-prep: deployment-cumin-3 2d02fdb4-8068-4d75-9be2-ac2ea715b8c4 [] deployment-prep: deployment-db11 369dfb8f-6311-4f99-85c8-cd9aa293d630 [] deployment-prep: deployment-deploy04 60461772-4536-4055-ab8e-9180c719cf25 [] deployment-prep: deployment-docker-charts01 d6b21175-b9f9-454a-afdc-6e647c40cd02 [] deployment-prep: deployment-docker-citoid02 a092d208-885b-40e4-9d27-998669e9ac11 [] deployment-prep: deployment-docker-cxserver02 b0ef7806-2ed0-4bbc-afbb-175a8ec1d85a [] deployment-prep: deployment-docker-mathoid02 52009c1a-5a0b-4d47-abf0-11efc090e692 [] deployment-prep: deployment-docker-mobileapps02 c6221ce4-48ac-4129-a984-eb7db83b0adb [] deployment-prep: deployment-docker-proton02 918c1510-917c-43f7-9829-d6f669a700c4 [] deployment-prep: deployment-docker-wikifunctions01 52cca4f5-9aeb-4a4f-90d0-b0f2d5e07cb7 [] deployment-prep: deployment-elastic09 11a817d0-3184-47be-a846-316ceb21221b [] deployment-prep: deployment-elastic10 5ef1f5ce-b1d9-48c4-9f85-c727f7d12851 [] deployment-prep: deployment-elastic11 9e94e6d5-a094-4ba3-8594-0cc1d3c4e315 [] deployment-prep: deployment-etcd02 c27abddd-8330-43db-80a9-dafebc76fc79 [] deployment-prep: deployment-etcd05 9e5e712f-eb60-4136-a66d-eb23a56d3349 [] deployment-prep: deployment-eventgate-4 2d94e48b-4e1b-4cf9-8b11-9d0d4bbc8efb [] deployment-prep: deployment-eventstreams-2 beed7e0d-4e7f-446f-a73c-60dce7ecff4f [] deployment-prep: deployment-ircd03 ea50001a-ca9b-44be-833f-2563d13e9779 [] deployment-prep: deployment-kafka-jumbo-5 3ceeda11-82e7-4fd6-9321-24713686318f [] deployment-prep: deployment-kafka-jumbo-8 1b4812a5-db65-4aa2-8adb-6af7333ae950 [] deployment-prep: deployment-kafka-jumbo-9 8c77502b-f887-4974-99b1-387c776605af [] deployment-prep: deployment-kafka-logging01 4f08a0ac-5810-47fd-941c-dbc883811f43 [] deployment-prep: deployment-kafka-main-5 b2eeb153-d31b-4c92-9c06-c64242bda3a0 [] deployment-prep: deployment-kafka-main-6 9178db63-751e-4727-8b9c-4b58f44cd214 [] deployment-prep: deployment-maps-master02 e170e0e7-226b-476e-9df0-4a48a53f4366 [] deployment-prep: deployment-memc11 0776b1c1-956f-46f6-a909-dca50a3d2f28 [] deployment-prep: deployment-memc12 f019dd93-b279-419d-a9fd-36e1256150fc [] deployment-prep: deployment-memc13 77aaac97-8184-4ab1-a8a4-947fca686420 [] deployment-prep: deployment-ms-be07 643b1616-eede-4859-8917-e10b4057126b [] deployment-prep: deployment-ms-be08 20f22f9d-5734-49b9-8a19-9908e588c92f [] deployment-prep: deployment-ms-fe04 7e6b3fec-b97a-4a32-b5c6-279244f78739 [] deployment-prep: deployment-mwlog02 56bc21a6-4fd4-4ea8-9cb1-2c9d9f170c15 [] deployment-prep: deployment-mwmaint03 69da7d16-942e-45d7-afbd-f01eb1d88d9f [] deployment-prep: deployment-mx03 8eb272dc-8006-4e93-a966-5035809324d9 [] deployment-prep: deployment-puppetdb05 7bd079eb-82f6-453a-9bf4-ddcf68057de3 [] deployment-prep: deployment-puppetserver-1 3192a8d0-0fda-4d24-a1b7-2e2dc20fbe0e [] deployment-prep: deployment-push-notifications02 c1994a14-51da-434a-941b-69e14dbcdaf9 [] deployment-prep: deployment-rdb01 0df5294e-2b0c-4b32-aa01-9774da28b01f [] deployment-prep: deployment-restbase05 0f8ff3ed-24ad-4dc8-ac30-6423901a66f1 [] deployment-prep: deployment-schema-3 b4da0c29-b7bb-40a8-9fbf-d54dc72d8257 [] deployment-prep: deployment-sessionstore06 716ce882-8593-4129-90d7-16adb99ae3d1 [] deployment-prep: deployment-shellbox01 1c094211-c3c9-43ef-8bd9-8f2255d85e5a [] deployment-prep: deployment-shellbox-video 6789abfe-3a7f-4d3a-bf32-0b1030c8be4a [] deployment-prep: deployment-urldownloader04 1d187238-9307-440f-b63f-3c3b6a34e239 [] deployment-prep: deployment-webperf21 ba5ee9c6-08e0-474f-8049-d31ec8bed047 [] deployment-prep: deployment-webperf22 a678753b-570c-47d9-86cf-bad88ab20bc6 [] deployment-prep: deployment-wikifeeds02 36458df2-70be-4666-aa09-eafef6b38fc2 [] deployment-prep: deployment-zookeeper-3 5794b8a9-a29f-46e3-8058-52188bade672 [] devportal: devportal-demo01 3fa16445-55ae-458c-a229-f3dd019de2af [] devtools: codesearch 5fca62da-0a71-4856-b58d-9566bdb52d14 [] devtools: deploy-1006 6c97c169-54f8-419b-a747-cd4e3b47f868 [] devtools: etherpad-bookworm 604af9b2-65f1-45bc-943a-52bed0012491 [] devtools: gitlab-runner-1002 7af5196c-795c-4d1a-bb11-15b3506cd16f [] devtools: gitlab-runner-1005 9b170998-ef69-4a3c-aa0f-35855107f691 [] devtools: phabricator-bullseye c2b2b2a1-140b-402f-be7a-50749c78bfc7 [] devtools: puppetmaster-1003 b7f828a5-8269-40f4-87bf-a78b3ce5f147 [] devtools: vrts-1003 a77ff416-995a-424f-a8bf-c7261e93b162 [] discourse: discuss-space 46e675f2-9115-4443-a91f-5c50b7850417 [] discourse: discuss-space-new ca3a1b1f-2a36-4e6f-83ad-d8bbb49b5862 [] download: download-03 02bcf69a-f8ae-4a31-b2ec-cd8e53193e65 [] duct: eb 23a8d36e-d5c5-4264-9d7b-a87d86bb99ac [] dumps: dumps-nfs-1 4761bc85-b4c9-4a88-ac13-8205a79f43d9 [] dwl: irc-bullseye 596d0df6-714c-4bc2-9049-1a459c1360ac [] dwl: taxonbota3 85578a0d-caad-40b4-8b5e-2126c220105b [] dwl: taxonkatbot3 60eb9364-03ca-4b2e-adef-3a155e03445c [] entity-detection: maps-workspace 7e758ab0-b51e-47b5-92ba-606bf19771ec [] etytree: etytree-b acfe0102-959d-46d5-bde9-3e03f8814df6 [] eventmetrics: prod03 5ebd0672-64c9-4dba-a687-b029767433ab [] extdist: extdist-06 472c3694-e68a-4c36-bce8-6517ab3bcc73 [] fastcci: fastcci-master 61e2acff-66b0-4f0d-a0cd-ce737feca69a [] fastcci: fastcci-worker1 b9352e93-fb87-4e32-b30c-55e475ade73a [] fastcci: fastcci-worker2 c3a88537-8f70-44e3-8f0e-9c0fe592429e [] fa-wp: chemistoid 38c1a830-be37-4472-8310-45f5ad5b277d [] fa-wp: tofawiki03 feaa04b8-27e0-4f78-abd9-a74a18fdecfd [] fr-tech-dev: payments e759a967-e322-4e3f-93c3-f546381a7b8d [] gitlab-runners: runner-1021 7ee0fcc7-43ea-4321-878e-a6922ac1236d [] gitlab-runners: runner-1022 cb4a87e4-264e-4c8f-8197-3efff54346de [] gitlab-runners: runner-1023 56e55a31-8d32-455e-b650-b7194e71d2fd [] gitlab-runners: runner-1024 921ddea1-0288-467f-8458-56860f383b93 [] gitlab-runners: runner-1025 6b8137a7-243e-4319-9e80-735095b8c30c [] gitlab-runners: runner-1026 329fc8c9-c51a-406d-9bad-9d88013b3e4b [] gitlab-runners: runner-1027 c3bdc48c-9ecf-48b3-bad9-67b6fc26cc77 [] gitlab-runners: runner-1028 b5b34987-d4ce-4f2b-a96b-b042d6e65623 [] gitlab-runners: runner-1029 39693d07-dad5-4ad0-a49c-e6fa468ae53b [] gitlab-runners: runner-1030 a9902e35-173f-4dbd-bce3-076e552bcd65 [] glams: glams-testing f9d47b22-bf9b-491c-b48d-dbbb56c487a9 [] glamwikidashboard: serviceapp 7aa37f8a-9e13-4ef2-8362-af49656c7e2d [] glamwikidashboard: webapp e78526c2-32c0-49f9-9b03-1ec49fff26be [] globaleducation: impact-visualizer-tool 7009df57-c7ec-45a7-a65e-da69eb018392 [] globaleducation: p-and-e-dashboard-redis b3c07919-318d-45a8-9c29-88cb3a59f56f [] globaleducation: peony-sidekiq-3 2962a54e-eaa3-4a46-a5ba-dc4773bf2fdd [] globaleducation: peony-sidekiq a78c5ea3-0c0e-4589-8ac9-8e24023a314f [] google-api-proxy: google-api-proxy-04 6bf5e226-050f-4307-8550-a3ef7e960ea3 [] hoiscript: Peony 6b32f902-7d6f-4ff3-a0ac-20f264d0c5de [] huggle: huggle-nfs-1 b74e3f99-5aee-4702-b501-7cc93cd8f4fc [] huggle: xmlrcs2 e840cdbd-349e-42f8-bf83-12507be6ebeb [] huma: web01 583ad29e-f9b3-47e7-8b83-0440bf72d7fa [] huma: worker01 cb20f298-f78e-4ae6-8752-9c32fb79a7cf [] iiab: iiab-med 79f31d08-d21f-4ef6-9cc0-7bf495cf28ed [] imagebulk: imagebulk 21cf9a6d-6abd-44e5-b12d-08b611643466 [] impactvisualizer: impactvisualizer 207f3ae3-9627-408a-8c07-bd6c78a56c71 [] impactvisualizer: replica-revision-tools 133429b1-2a3d-477a-ba3a-006939ba138f [] incubator: incubator-1 d7c826ce-1b23-4b63-bf2c-280b9396644b [] integration: integration-agent-docker-1040 42d3d9a4-5d2d-433f-8b3f-c8fdb2a24a3c [] integration: integration-agent-docker-1041 db4ae352-bbc3-450e-b4bb-1bb95f186072 [] integration: integration-agent-docker-1042 426d1a55-e8ce-4332-9e3c-3a52f47b36af [] integration: integration-agent-docker-1043 4c87de46-8ea5-476f-812f-4e627dbd6671 [] integration: integration-agent-docker-1044 88c795b4-72ee-4b82-b399-04d3457f2ce1 [] integration: integration-agent-docker-1045 d0e8fb1b-ec89-4934-86cd-6118db1219bc [] integration: integration-agent-docker-1046 270b6533-dc99-4e5d-a642-c61138b11891 [] integration: integration-agent-docker-1047 4c9a5ffd-f322-4b14-8f71-a87a18f54b09 [] integration: integration-agent-docker-1048 ab9080c3-fbcd-4e5f-90f4-d9669b9658de [] integration: integration-agent-docker-1049 80924e5c-f001-49f4-8043-6623af917151 [] integration: integration-agent-docker-1050 cfdbd9d9-6cbe-4516-89dc-b19f0e210f2d [] integration: integration-agent-docker-1051 ecbc72ca-601a-4ee5-b590-10606d64886a [] integration: integration-agent-docker-1052 7f7fe96a-c25f-4116-a79a-c77f1a88ca33 [] integration: integration-agent-docker-1053 14789ac1-bc06-4677-9bb0-66c16c887427 [] integration: integration-agent-docker-1054 702b9925-099e-4ecc-9d67-35865affa64d [] integration: integration-agent-docker-1055 6616fcbf-a49e-4e03-b735-84d31b535c08 [] integration: integration-agent-docker-1056 cc9e0883-0282-4f94-8e39-9c5ebe5ef805 [] integration: integration-agent-docker-1057 5c67ffe7-9c5b-4790-92b8-ca8299613ac0 [] integration: integration-agent-pkgbuilder-1003 a08a80f5-1b45-4aba-8945-cfc080a4c940 [] integration: integration-agent-pkgbuilder-1004 277f302a-b2fa-4635-8c88-0f8d6e9971f4 [] integration: integration-agent-puppet-docker-1003 96cf9ddc-daa3-4c9f-8c21-cdd58e95973e [] integration: integration-agent-qemu-1003 aa7075c2-f321-4817-a53a-ce553ecb6904 [] integration: integration-cumin 80f8822f-c0d9-427a-b743-b9185f58c0eb [] integration: integration-puppetserver-01 a2ce4233-268a-45ac-b542-ab6fc4e660a6 [] ipoidopensearch: ipoidopensearch 41b83b6d-344e-4b13-b0d7-a8a7e8b72dfe [] isa: m2c f8b9e023-248d-49a7-bf38-2dd448c48d33 [] k8splay: abogott-control-1 e470f1de-2f84-4068-981e-4af29685f04a [] k8splay: abogott-worker-2 e28a75a0-daf2-453b-8efc-fe88ae819470 [] k8splay: dzahn-bookworm 5ec028f1-3fcb-4774-9984-b3bc3f7a163d [] language: cxserver 9e0f89e0-a177-4a2a-a677-32e888b6e37f [] language: language-cx 9a857e61-052c-4025-a4f7-e42c31edd3a3 [] language: language-mleb-legacy a44b9b31-a11f-4594-b9f2-170ba2bfb4eb [] language: language-mleb-master 1c0ac8ab-1c86-4ad6-a65f-7802984c5ae4 [] language: language-sx2 a63a8815-e397-4b8d-9ec6-98fc3f380ef7 [] language: mleb-stable 78fae2d3-dc53-4242-9e18-3edacc02786f [] language: recommend 3e3fac3d-be77-4f50-8b18-9d7616c8fa58 [] language: translate 15ed27d5-cc31-462a-97b5-68601d3e88d7 [] ldap-dev: dcl-dev1 80bac244-e231-4390-9de9-79dc71ac0827 [] library-upgrader: libup-runner08 9a99163d-35a6-4596-a559-9014e72e0379 [] library-upgrader: libup-web02 53f8c1f4-0c85-407d-806f-47b585852cd7 [] linkwatcher: coibot-2 e08f00f6-5a4c-4981-8ee1-75e773204742 [] linkwatcher: linkwatcher-db 8ff173eb-163e-47bf-a429-a114d7affce8 [] linkwatcher: liwa3-2 ce9426cb-923e-4af7-a703-f81d9b3e6149 [] linkwatcher: xlinkbot-2 168548e6-4a9f-4faa-a109-17c90b6cb421 [] logging: logging-alert-01 a1141332-0400-4a87-ad7e-24564676e48a [] logging: logging-logstash-02 15a048f8-ab89-43ff-80be-63cf5b5691fa [] logging: logging-logstash-03 14abc0a9-ef08-4e73-a54b-8d91c1bbc752 [] logging: logging-opensearch-ssd-03 f502126e-1aa7-4f02-a047-bed58688e30c [] lta-tracker: bots2 27acf162-300a-4d92-a293-5b2d34e5c4b5 [] lta-tracker: prod 2b56c575-08a5-4def-87cb-bee5bd43e4f9 [] mailman: mailman1 187649b3-0313-46d4-bbdf-34b08d5c3d48 [] maps: maps-warper4 d3521b71-1096-4fb5-8a34-33ae6191965c [] mariadbtest: cumin-1 3776d295-b63e-4766-ac60-a7027821d11a [] mariadbtest: db-1 bafb528e-0490-4ce6-9505-a0526f95782f [] mariadbtest: db-2 68aa2f4b-a14d-4d82-b1f4-e6a0f1ec2d1e [] mariadbtest: mariadbcompiler e50dcf0c-9bdb-4a45-b337-9e46e3ba2e42 [] mariadbtest: test-with bfd02baa-da18-41c0-b06c-f6028bb5440e [] mariadbtest: test-wo 1cd86198-ce89-4c46-8218-20313d467f4b [] masz: web02 b43ae478-d62c-43c1-ad8f-eb09ca6f9f86 [] math: math24 0235bb05-e38c-44f6-93ac-13532c32ffff [] math: math-docker-2 372adf1a-3379-4fac-b2f3-e05c48ba869e [] matrix: synapse01 1fa9f0d9-42e8-4273-bdb1-a7d49998c13f [] mdwikioffline: mdwiki 60dff349-faea-4de9-b4d0-b776f43f2e5a [] media-streaming: media-streaming de2f0e88-ca81-4345-b445-9036de1740fb [] metricsinfra: metricsinfra-grafana-1 ae59a32d-c913-4b75-bf08-e598a43a2892 [] metricsinfra: metricsinfra-haproxy-2 46dfccb9-f74b-4b99-934d-f724b1c6a63b [] metricsinfra: metricsinfra-prometheus-2 7cb371bb-a53a-4e65-a1cf-f1a8264a9166 [] metricsinfra: metricsinfra-thanos-fe-1 b67ceb26-97c1-402e-9c7a-7c6d52b2f119 [] mwcli: gitlab-runner-addshore-1016 37e23659-4516-4bd8-a9be-4cc55def5560 [] mwstake: mwstake 77e0ad73-61dd-4a9d-a0c8-fc89e30a858b [] netops-clab: netops-clab de3c8ae8-9e09-4ece-9c19-07c39365b239 [] o11y: deployment-server-phi 5899163e-bbfb-474e-8a9a-2dae5006de22 [] o11y: phi-alert-01 98e6d07d-dd64-4083-8fab-f180e11dc402 [] o11y: phi-arclamp-01 cd686956-e65a-491f-9fc0-7bb216c61fa9 [] o11y: phi-grafana-01 8d5dba20-52d6-47f2-b26e-73531401d813 [] o11y: phi-kafka-01 e0878f27-bcff-48b9-a9c3-8af797e92622 [] o11y: phi-kafkamon-01 1b7de1cd-0857-425a-84d2-a4af29f6e982 [] o11y: phi-lb-01 562d3627-c869-4255-8818-90cc6d059ad4 [] o11y: phi-logs-be-01 940e0318-604f-4a88-ba11-e0f8ee0e0cdc [] o11y: phi-logs-fe-01 0c57ca62-098d-4f99-900a-d6cc3d88cb6d [] o11y: phi-mwlog-01 1802b911-6aa9-4142-b54d-a5b542f504e8 [] o11y: phi-pki-01 6383bd75-63cf-4703-b3a7-acb84a4cb93d [] o11y: phi-prometheus-01 d7ec787a-55f4-42a0-8d03-00efbccc8d21 [] o11y: phi-prometheus-02 871a7169-2748-41ec-a3e9-6b13a81ec526 [] o11y: phi-puppet-01 e964291c-48d8-4d44-89dd-24e4d49d7f97 [] o11y: phi-puppetdb-01 64149a5f-eeea-4978-9921-17de05c88dd6 [] o11y: phi-syslog-01 5393926d-b5cb-4132-9bf1-06f0a89d1d9c [] o11y: phi-titan-01 2d729afd-8290-4744-ab1d-57297ca6bbf6 [] o11y: phi-webperf-01 e5ed77fc-1077-4129-80b3-120c570c8edd [] ocrtoy: ocrtoy-web-bullseye 347d3b08-cd71-46f2-8028-ec5581719114 [] onfire: klaxon-keith 44ca15de-5737-4daf-8418-bb0068e106ed [] openvas: openvasv1 a236086b-56a7-42f5-ad04-a99dd10cafaa [] osmit: osmit-estratti 457f53c3-fbce-4b3d-ab72-bbb08c2716b7 [] osmit: osmit-umap2 1e476394-4ed9-439e-aed4-053486653df7 [] owidm: owidm-dev-1 c1f67eec-fff5-4ea1-9c1a-450b2702c206 [] owidm: owidm-prod-1 93eb7e45-7bd5-461b-93b1-e0637e947de0 [] packaging: cas-build ebfaa640-88ba-4bdc-a7d6-f35cd80a14fc [] packaging: packager-etherpad01 05a87de9-51e4-441c-bed8-03372ac90231 [] paws: bastion 9c7625e2-2dd0-4289-a934-f617fbb7999b [] paws: paws-nfs-1 1b001a01-0a37-4d81-9dd6-1339b6391f37 [] pixel: beta 2f8314c8-6d73-4462-ae6b-1d8ea7fc5e68 [] pixel: production 7af72145-4bdb-4a19-b960-f428eedfcb7b [] pki: pki-db 3788dd45-8e96-471f-a4f5-4c788c12601b [] pki: pki-intermediate f56e0ae1-b26f-43d8-8e9e-2e7fb424ce0b [] pki: pki-pm 22e7671e-c144-4508-afe0-f0e24603fa69 [] pki: pki-proxy 7081de14-d7f5-44ca-b82e-6b99af667696 [] pki: pki-puppetserver-1 1e528d51-9710-4047-ba0b-a58e9e53ae35 [] pki: pki-root edfc4cbd-dd02-41f2-8732-c48ef55e9c67 [] pki: pki-test01 3e6c31d7-b4db-4a5f-a610-a74d0013f631 [] pluggableauth: survey 8cf4a071-f4c1-4312-b635-926e0742ae5c [] pluggableauth: wikifarm 9f2a71c3-5683-4c30-b28d-617396fc3e3d [] procbot: bastion 102ecf69-5699-49bc-92ec-17b4823cbcb0 [] project-proxy: maps-proxy-03 cbb613c3-9480-4e96-a83e-90560205657f [] project-proxy: maps-proxy-04 d63dd9dc-a2f9-434e-b315-5d9d1d4f7c9a [] project-proxy: project-proxy-acme-chief-02 fd43cf27-3b19-4035-8047-7529c69b3e53 [] project-proxy: project-proxy-puppetserver-1 e8a89ddc-9129-42c4-bac5-1034de4ad90d [] project-proxy: proxy-04 24219963-e446-4c89-8ad7-30e8c4c69168 [] puppet-diffs: pcc-worker1004 a0d371be-dafb-4228-89c0-fc25b8d7430e [] puppet-diffs: pcc-worker1006 f9c9f01d-b1fd-4e64-a521-522fc767e604 [] puppet-diffs: pcc-worker1009 b02c4866-e0da-4029-b7ca-73252b5492c3 [] puppet-diffs: pcc-worker1014 72986a86-4401-4d7c-9730-5a71d7d22624 [] puppet-diffs: pcc-worker1015 a4871202-b93d-47e5-8b5d-eba47b274c5f [] puppet-diffs: pcc-worker1016 7589c350-58c4-4064-ad91-418e44ef5432 [] qrank: webserver 6efd77f9-2725-4a22-a898-625e3123bb0b [] quarry: k8s-nfs c18ab12b-fa18-4a29-a074-40535930f6b0 [] quarry: quarry-127a-g4ndvpkr5sro-node-0 8b68b650-19d3-421c-975f-7f98a47b5d2c [] quarry: quarry-127a-g4ndvpkr5sro-node-1 c86b3288-aff1-40a5-b619-887bdcefca44 [] quarry: quarry-bastion bec16bd4-988d-46b9-8b73-d044a4392a92 [] reading-web-staging: simple-summary-server 42121024-526f-497f-9001-12925c59fb1e [] reading-web-staging: skin24 3196c742-86ba-4308-b5a1-04d4bace5691 [] reading-web-staging: tarsier d99bd61e-094e-4a97-b856-96cad37fa905 [] reading-web-staging: wikitalky a43d6054-daf8-4049-b445-4952c98ddced [] recommendation-api: article-desc-deprecation 7bbd7cb5-8784-4091-aa0b-3058303695c1 [] recommendation-api: gapfinder-deprecation 1fdcc8e1-2274-4232-87d3-924a60800744 [] recommendation-api: search-referrals 24d2da5a-36b2-46c7-9217-9b5dc0d2a022 [] redirects: redirects-nginx03 dfbc6968-d803-4885-a34e-ba53cda922e7 [] research-collaborations-api: citationq b038356d-53f2-456d-9654-564261db0395 [] research-collaborations-api: edit-types 5f26c230-c05a-4f3c-ad4c-d4aae19f1c5e [] research-collaborations-api: fact-checkign-new f2f0790b-948d-4523-bfa5-a332da588aec [] research-collaborations-api: kitools 7da824f3-c9f3-4460-b9b2-2a894268a7ca [] research-collaborations-api: wikinav-bookworm 774a256b-2b8f-4384-93e7-53d368ff9a96 [] search: airflow 69825ca0-1fe2-4960-9c1f-24b1425ae85d [] search: cirrus-integ03 28e597a1-769e-48df-be12-73dcf53ea658 [] search: opensearch01 68a499c0-54a6-445c-af12-8e5265b10bee [] security-tools: security-dashboard 4bd8f0ba-26e3-440f-a972-f3218f2be97f [] signwriting: signwriting-swis-2022 e4bb9712-5ac3-4e57-8aa7-58754aa5216c [] signwriting: signwriting-swserver-2022 c778195a-b439-4c32-843b-40db2c6404b1 [] soweego: soweego b958511a-10cb-4e62-bdbb-6da5013dd62f [] spacemedia: glitchtip 3717d09b-ca52-4b92-8126-3720f4930251 [] spi-tools: bullseye ffae23a6-47bf-4c76-bfd2-d6b7d2cf495b [] spi-tools: opensearch2 e56b810b-6a74-41cf-8aa0-5a5fddcb2014 [] statanalyser: statanalyser f78be803-6d05-43cc-9b40-a8748ff56204 [] striker: striker-docker-02 5acbe429-549d-4d2a-889a-ed1bb841ee43 [] suggestbot: suggestbot-02 f5ff0e3c-82ea-428d-a263-9044a0419249 [] superset: superset-bastion 05a0a9b8-d56d-4117-bd40-f747869397f0 [] swift: dp-cssndr-puppet-01 2848e015-24ef-4061-b7a2-41401cffbf0a [] terraform: tf-puppetserver-1 99c2a647-36ef-40d5-9c21-8a20f437daff [] terraform: tf-registry-2 7946a9ea-136a-4223-8996-2e3a0334e3d8 [] testlabs: abogott-puppetclient1 6d344a91-80e8-48b0-b334-9bb77b153116 [] testlabs: abogott-puppetserver 74bca50a-fafc-483f-ad29-a68b3960d73a [] testlabs: kolla-test-3 92c89dd1-0cd3-4ca7-a7e3-09d4c3673814 [] testlabs: rook-tf 3ceb9b8d-a544-48f8-86ae-986ea938d7ed [] testlabs: slavina-k8s-workshop 8fc0eed6-8f20-4916-a81d-fb82b2d704cf [] testlabs: testlabs-nfs-1 14695d95-e47d-4b27-9023-20a38f07d11f [] text-to-speech: monitor 35d1d34c-7bc7-443b-a0c6-922de7368549 [] teyora: teyora 68b3e7b9-11ff-4223-a20b-dd19664b88f5 [] teyora: ty-analytics 4ae3bca1-7ea7-489c-9433-d0ebc2652e18 [] tofuinfratest: tf-bastion 3c385a09-9d18-42cc-965b-5d6cee9d005c [] toolsbeta: toolsbeta-bastion-6 33e430fb-f679-41ce-8ee6-4db1e812454c [] toolsbeta: toolsbeta-cumin-1 868d3dca-3e5c-4089-89a9-2c7e756c3e31 [] toolsbeta: toolsbeta-docker-imagebuilder-2 c07c4b31-fa95-4f18-927b-a17cf1003ae5 [] toolsbeta: toolsbeta-harbor-1 caf923d3-a42b-49b8-b861-f20996ab4ab9 [] toolsbeta: toolsbeta-legacy-redirector-2 ca99f9d1-a806-4eee-ab96-e4712abc611b [] toolsbeta: toolsbeta-mail-2 1cd98b82-6c77-465d-8dcd-f0c20e6d7789 [] toolsbeta: toolsbeta-prometheus-1 d110734e-aa20-404a-9f1e-a58c266f9fc5 [] toolsbeta: toolsbeta-proxy-5 1870a647-224b-4b60-b16f-64714e15cf7b [] toolsbeta: toolsbeta-proxy-6 64e17166-af72-4aa6-a0b7-544a12569655 [] toolsbeta: toolsbeta-puppetdb-03 56691327-d836-4f78-a5fd-93bdef49bfa9 [] toolsbeta: toolsbeta-redis-4 44007ae1-c81b-4cc8-b733-536daf5c14e6 [] toolsbeta: toolsbeta-redis-5 f794f6fd-59fa-4fd1-9179-f348e2a0c9ab [] toolsbeta: toolsbeta-redis-6 019519e7-2169-4615-8985-424af8d0e0d7 [] toolsbeta: toolsbeta-static-2 13f3646e-000e-4cbb-a643-54f73eecae4e [] toolsbeta: toolsbeta-test-k8s-control-10 7d35bbd7-cc41-48b3-87a5-35604980c40d [] toolsbeta: toolsbeta-test-k8s-control-11 44513a1b-807b-47cb-9f47-8ae343fe5f82 [] toolsbeta: toolsbeta-test-k8s-control-12 652f13e1-1c7f-46cd-9ca1-d7bebdc5abba [] toolsbeta: toolsbeta-test-k8s-haproxy-5 b65d430f-8873-48b4-9a08-681ab0552739 [] toolsbeta: toolsbeta-test-k8s-haproxy-6 8fbe70cd-a471-4f8b-b3e8-0136c20e8a3e [] toolsbeta: toolsbeta-test-k8s-ingress-10 5d4584f9-559c-4d3e-9af5-6542200b5158 [] toolsbeta: toolsbeta-test-k8s-ingress-11 5130e84c-f8fc-4e02-8c93-9412dab3f630 [] toolsbeta: toolsbeta-test-k8s-ingress-9 72561491-3b1e-437c-878d-e1db110728f3 [] toolsbeta: toolsbeta-test-k8s-worker-12 e5f37100-ebe9-4238-98ae-4869237baf36 [] toolsbeta: toolsbeta-test-k8s-worker-nfs-10 8a0ca656-8179-4544-9154-7d11b8941899 [] toolsbeta: toolsbeta-test-k8s-worker-nfs-5 60e69fe7-f6dd-48d4-85f6-3c13a946c2da [] toolsbeta: toolsbeta-test-k8s-worker-nfs-7 cd12b691-19cb-4e28-a1f1-433e53378f8e [] toolsbeta: toolsbeta-test-k8s-worker-nfs-8 93848139-6952-4edc-a927-9f207fab6b35 [] tools: tools-acme-chief-3 452dc8d3-6ee0-412c-90ba-66f21f9d09c1 [] tools: tools-acme-chief-4 2cfcfc76-8443-4b04-b129-ba381e6fbf22 [] tools: tools-checker-5 f0936015-b1ef-4b8c-ac6e-74766ed882ce [] tools: tools-cumin-1 5c8b0a31-4074-4629-a996-b4c156cdaac6 [] tools: tools-db-4 230c30f1-37a1-42a7-a829-4369448b73f5 [] tools: tools-db-5 51a560c1-d6dd-42b9-bec9-bd2b2504f2d8 [] tools: tools-harbor-1 243f3a30-eb5f-4726-9862-bcb1a35e267b [] tools: tools-k8s-control-7 67dbc73c-a7f8-40e4-96ef-ee838766ecff [] tools: tools-k8s-control-8 1d0839d5-0a11-4ed2-83b8-b0c15c62a5ae [] tools: tools-k8s-control-9 c195fc23-61a1-4093-9c9c-02f72e793380 [] tools: tools-k8s-haproxy-5 1de296d9-e469-47ab-a1ce-eb08c38d05b5 [] tools: tools-k8s-haproxy-6 469f0822-42df-4268-9e75-32e6973bf86e [] tools: tools-k8s-ingress-7 a96f04aa-9200-4b1a-86a8-3bfb836843bc [] tools: tools-k8s-ingress-8 54a3d128-ddfc-4346-bf84-57df94409fd7 [] tools: tools-k8s-ingress-9 959a47f2-87b1-43f3-bce5-f302fa3d1a3a [] tools: tools-k8s-worker-102 1bac7184-6f29-43b2-a686-1eba60ffbfed [] tools: tools-k8s-worker-103 1f64b6ab-4318-48e5-b885-c80151d4ab8f [] tools: tools-k8s-worker-105 0a3ebade-abf5-496e-88d5-ca29ceb52965 [] tools: tools-k8s-worker-106 eec8ebe0-64bf-40c1-b470-8e017b8e20e9 [] tools: tools-k8s-worker-107 da629018-4983-441c-b83d-7ecf90b6dd48 [] tools: tools-k8s-worker-108 3019cf57-1301-46f0-97f7-234b006e8be6 [] tools: tools-k8s-worker-nfs-10 7a497017-e462-473b-bdf5-5798f08b3179 [] tools: tools-k8s-worker-nfs-11 91080d53-f5a6-4d96-b622-722405de8982 [] tools: tools-k8s-worker-nfs-12 81a33d19-55cf-407c-b888-5b06438a095f [] tools: tools-k8s-worker-nfs-13 8155a7ef-5889-421a-a1fe-d43c00d60a0e [] tools: tools-k8s-worker-nfs-14 524da4e8-c4f6-4ed1-b18a-5bbcc51900ac [] tools: tools-k8s-worker-nfs-17 00aaf7e9-b8bb-489f-965a-bb99ecc050e2 [] tools: tools-k8s-worker-nfs-1 7b355092-f5e4-4be0-af94-b1307983c8e9 [] tools: tools-k8s-worker-nfs-24 77140f83-1a12-43b7-9e47-0e779503a525 [] tools: tools-k8s-worker-nfs-32 cab36feb-ef28-4f5f-88db-83270d181138 [] tools: tools-k8s-worker-nfs-3 5312e2f8-035c-464d-903e-79670197ab8f [] tools: tools-k8s-worker-nfs-39 8fb68878-67fe-4118-bb08-2f88b6e5f095 [] tools: tools-k8s-worker-nfs-40 728ae870-9430-4cd9-9fe3-d8138b97169b [] tools: tools-k8s-worker-nfs-43 18dc79d9-df55-4bf6-8bcf-5f40fb22aecd [] tools: tools-k8s-worker-nfs-45 62a59163-64c4-4295-bac0-f7f7aaabadf7 [] tools: tools-k8s-worker-nfs-46 1842d2b7-ca46-4c16-857b-8d588d5df6c6 [] tools: tools-k8s-worker-nfs-47 1423479b-afaa-4e16-b970-f4fb81db76f0 [] tools: tools-k8s-worker-nfs-50 3c99f41d-8a93-4347-8bc1-3ad5c63f55b1 [] tools: tools-k8s-worker-nfs-54 761089d7-d849-4b5d-8f8e-7368738f70dd [] tools: tools-k8s-worker-nfs-66 f9015696-9010-4a1c-a7cf-9bc8e8098df4 [] tools: tools-k8s-worker-nfs-71 e06b79a7-990b-4142-844e-5e826f3d4569 [] tools: tools-k8s-worker-nfs-74 11266010-8581-4750-9375-03fd1efb9e98 [] tools: tools-k8s-worker-nfs-9 33bfa2d5-31c5-4fe0-9a77-aaefd42eef1d [] tools: tools-legacy-redirector-2 7db5fc0e-34d2-495e-8045-8602639fe2bf [] tools: tools-mail-4 1484e4c9-0e93-4476-8882-56a7da42ecaf [] tools: tools-package-builder-04 62045cf9-59ed-44b9-a268-1c9f171b5aae [] tools: tools-proxy-7 31457fd1-ba1b-4231-ac7b-5e1df1d7f458 [] tools: tools-proxy-8 24d99fcf-6a44-4321-a653-00f7e839c479 [] tools: tools-puppetdb-2 94166aa7-59e0-4f8c-9bec-e8b018fda855 [] tools: tools-redis-5 5d256217-56b6-4e10-b4a8-422738b6d244 [] tools: tools-redis-6 10457456-a444-49e9-9468-ffbdf5a73703 [] tools: tools-redis-7 5ee8dd81-2224-4a6a-bdbe-b0a498f101e5 [] traffic: traffic-acmechief01 b49ed32f-fac3-4c19-b031-abb3f3bc2e36 [] traffic: traffic-cache-bookworm 4bca356d-86cc-4e10-a7a6-db574cf706b3 [] traffic: traffic-cache-bullseye ab5877a3-5184-485f-9d5a-afec6fa6b4cf [] traffic: traffic-cache-text-bullseye e634e20e-a0c3-4332-a1dc-0a75812a03a8 [] traffic: traffic-cache-upload-bullseye 3c766c63-f92a-486c-900c-1f752619332b [] trove: maps-test-2 b615abe0-9cc2-4cee-a224-c1f0b90e7957 [] trove: quarry-db-02 4b932658-2b4d-4157-b62e-1e73faaed8aa [] trove: quarry-k8s d277044f-a5db-4939-b1e4-5a142116f9d0 [] twl: glitchtip-twl 3927f0a3-d8f8-488e-a00b-27482202e737 [] twl: prod0 56cd3c88-7fcf-4bfb-a192-a86ba04e44a0 [] twl: staging a04ce916-1310-49bf-ac50-e5aca7738a5e [] twl: twl-nfs-1 64e0fcfb-4f8b-4912-ab79-83984205d016 [] usdtest: primary-app 6356dbde-37f8-40fa-ae3e-26c5913702be [] utrs: utrs-alpha 5aed0a3d-c48d-4c36-b9d5-f216de0dbfc1 [] utrs: utrs-database 5203247c-7191-4bde-88d4-7dea51a31920 [] videocuttool: beta-videocuttool-bookworm 565fe5fd-1b53-46bc-a5ec-410ab5e65537 [] video: encoding02 9a9b2610-e1ac-4dda-8309-2dc816f78dab [] video: encoding04 8ab8234a-5e85-4de6-a42c-031c7f05850d [] video: video-dev-bookworm f17b4b49-c1f0-43aa-8cd4-16a07288f704 [] video: video-nfs-1 6aa18f0e-d275-4c05-ab9f-65da1b7a7c3b [] video: video-redis-bookworm 02961b09-1520-4b10-b6d6-6118133cba20 [] videowiki: db-instance2 79f8e042-3bbd-4777-9d9d-509363b436a1 [] videowiki: videowiki-app-1 577a10c0-dfce-4bf4-8452-db0b1c68b248 [] videowiki: videowiki-db-1 2acb2154-ec20-4649-b7bb-6d619676e4ad [] vuessr: prototype2 cdb5e683-ddc7-4714-9d3d-951cf01af9ef [] wcdo: wcdo02 d28c57b4-2c75-4e24-a900-52bcd9c2e402 [] webperformancetest: crux-metrics 241f3730-4e8d-48c9-90f7-8c67e271d273 [] webperformancetest: queue b54d0173-24e7-426a-97db-7d9ca9489009 [] webperf: wikidiff2-demo 8cd85dcf-5ef9-4f6e-8428-e0312ff0ccb0 [] wikiapiary: scripts d2e8dc6a-0d72-4e84-99f1-ba669b12cf09 [] wikibase-registry: redir1 fb6a81ce-0337-4133-8928-47c2b5bb158d [] wikicite: wikicite-refsdb-proc-1 a11237f6-9436-4ad8-a8f9-0af4e020663c [] wikidata-dev: special-new-lexeme-testing 4f62099c-b9bc-4094-8c4f-db435781fdf7 [] wikidata-dev: wikidata-icinga-2024 504cb696-f783-4806-9e95-b1b6c2d88096 [] wikidata-reconciliation: wikidata-reconciliation 692086ce-c855-4f66-8b92-58426516a880 [] wikidocumentaries: hupu2 c13fc11c-3856-4588-96b9-8473878d752d [] wikidumpparse: wikidumpparse-nfs-1 fea0bea6-8b82-434f-849d-b837f98ff9cc [] wikifunctions: piglet 6143e81e-b683-4b0c-ac84-1ecc7a9b0cf0 [] wikifunctions: porky 11c29cdd-fd9b-455a-90d3-9593f17f44b7 [] wikilink: glitchtip 258ed0d6-41a5-4b96-b057-6ca71037b3f7 [] wikimania-mautic: mautic-web d92e8df1-0e53-4607-ab9c-1e59de1fba97 [] wikipeoplestats: wikipeoplestats-prod01 c2f528c1-6668-4619-91f9-b6ef53d3d8a3 [] wikiqlever: puppetmaster 57d26b02-dcb3-447e-b2eb-ac667852932c [] wikiqlever: qlever1 b2c96f28-03bb-4ff4-ab83-f79cc943d31d [] wikisource: ia-upload-test2 f6e22480-0e2c-4cd3-8d67-08250ab32709 [] wikisource: kraken-ocr 62e585db-7abf-4a0a-b076-595cdc0521d1 [] wikisource: ocr-prod02 22aa1613-9328-43f0-a0a9-f9bb48dcc635 [] wikisource: ocr-test02 d9acbf41-02da-4604-ac9d-93e1fe769b66 [] wikisource: wsexport-test 1328e006-5595-417c-864c-f771f027c4a1 [] wikisp: diana 4a74b89d-185e-41ca-82b3-fd121b477c47 [] wikispeech: demo-wiki b010ef0b-475a-4c6d-aa87-3f33efac317d [] wikispeech: tts-dev 24adb48b-ee87-4399-8036-0d0daefcb99d [] wikispore: owmf-test 0b1e12a4-dfff-42ae-84ca-0754959ffbb3 [] wikispore: wiki-nyc-owmf f6074ec9-28eb-4653-ba84-e7d735e3122c [] wikispore: wikispore-prod-02 5a312470-f9be-45ed-848b-6406211b3e82 [] wikispore: wikispore-test-bullseye 30dd5c6d-0444-4ed0-8386-fe232e0588b5 [] wikistats: wikistats-bookworm cf582bc8-0c2f-4ec5-8064-2cad2110b2ad [] wikiwho: wikiwho01 1b0a01e4-e8cb-4a68-ad6c-306b9869db05 [] wlm-it-visual: wlmitvisual01 1931dcad-42dd-4cdd-8246-0d13bdb38ad1 [] wm-bot: wm-bot-pg2-bookworm 63b82d38-3026-408f-8dcc-0ecbd1a3c870 [] wmcs-uptime: test-01 b96b8d02-164b-4d08-9046-1fd2b467a564 [] wmde-techwishes-survey: beta 664e04b3-34b7-4b87-9296-bfb1cbb5f872 [] wmf-research-tools: cloud-modeling dce8372c-fa22-4f73-a0d5-a855e342810e [] wmf-research-tools: dp-pageview-visualization c5688475-f082-460b-9c3f-81ebf1ff78cb [] wmf-research-tools: geo-provenance 3351eecb-e2ca-4889-a5cf-dad5b40add0f [] wmf-research-tools: knowledge-gap-index-tool 948c8c55-6fa8-45f7-9e3e-f3d923f2172c [] wmf-research-tools: mwparserfromhtml 609f5c13-499a-4fad-8b1d-6bf70e6e3b36 [] wmf-research-tools: outlink-model-embeddings 00d1b8e5-1211-4ec1-b975-0a5955fdcee7 [] wmf-research-tools: person-topic-test 9b9ecd8f-1ff1-4dec-a48b-722c15cf0bde [] wmf-research-tools: quality-article 90239a84-0dd4-4414-a6ea-5f72d98846d5 [] wmf-research-tools: quality-bulk-prototype 128d77c8-8c86-42c9-a3d5-6c94c1b8b860 [] wmf-research-tools: reader-embedding 7e5361b4-f195-4316-ada1-a156da2c8b69 [] wmf-research-tools: research-api-template 3ded4501-74a9-4752-9fdb-7d8f03ee9742 [] wmf-research-tools: section-ranker 507f6cad-3313-49c8-b4aa-a7ff7f8bc3bb [] wmf-research-tools: topic-model-prototype c7354620-5323-4888-9fcf-e5d55d263d4d [] wmf-research-tools: vector-search 3684f788-7edd-4e5d-a7aa-240d54f9c4a2 [] wmf-research-tools: wikidata-quality 58ca6c1c-e6b0-4963-8687-f3d81776fab6 [] wmf-research-tools: wiki-region-test 1f42bdbd-6dae-4387-b81b-f79dda18c9a4 [] xtools: xtools-dev06 1edf4d4e-fced-4815-b945-ba94b20f6011 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

2 4

[Cloud-announce] cloud-vps/toolforge network maintenance Tuesday, 2025-02-04 13:00 UTC
by Andrew Bogott 04 Feb '25

04 Feb '25

Next Tuesday we'll be replacing[0] the cloud gateway nodes that serve the entire cloud-vps and toolforge network. If all goes well this will only cause a quick connection reset but we're scheduling a window of potential downtime in case things do not go well. This work is in response to hardware errors that have been appearing on one of the existing gateway hosts[1]. -Andrew + the WMCS team [0] https://phabricator.wikimedia.org/T382356 [1] https://phabricator.wikimedia.org/T382220 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

python 3.11?
by Roy Smith 22 Jan '25

22 Jan '25

What's the right --image argument for toolforge-jobs to get a python3.11 backend? I'm getting: Value error, No such image 'tf-python311' when I do: toolforge-jobs run add-tags-en --command $jobdir/add-tags-en.bash --image tf-python311--schedule "*/5 * * * *"

4 3

[Cloud-announce] VM reboots coming Tuesday, 2024-01-20
by Andrew Bogott 21 Jan '25

21 Jan '25

Due to a latent configuration error, many VMs need to be rebooted. The affected VMs are working fine now, but if restarted they will start with an incorrect block storage config and be unable to launch, requiring manual repair. I am planning to migrate + reboot all the VMs listed next Tuesday, which will rebuild and repair the broken config. If you find one of these VMs in an inconsistent state in the meantime, please contact a WMCS root and refer them to the related task[0] which explains the somewhat tedious process of revival. Some of the VMs to be rebooted are nfs servers, which may cause service interruptions for projects relying on those servers. Specifically: 1b001a01-0a37-4d81-9dd6-1339b6391f37 paws-nfs-1.paws c18ab12b-fa18-4a29-a074-40535930f6b0 k8s-nfs.quarry a1cb92ab-6083-4465-81a9-a283918f13eb toolsbeta-nfs-3.toolsbeta cf6dc802-f95c-48c1-a07b-8791185a5e45 fastcci-nfs-1.fastcci The rest of the VMs that need reboots are: 4b932658-2b4d-4157-b62e-1e73faaed8aa quarry-db-02.trove 14be24bb-a36b-4627-9254-d02a0987db83 data.wikipathways 14be24bb-a36b-4627-9254-d02a0987db83 data.wikipathways 7dea739e-7f9a-48ed-8169-e88300c77092 gitlab-runner-1003.devtools 9aa7e1f6-1e53-4c54-814a-1c6e7b8e55bb dannyb.wildcat 07543105-1336-4f7f-ae04-5c7989a0bfb5 larynx-be-01.text-to-speech 1bff1756-ce9a-40b6-8783-a01910dff1b6 storage.qrank b26f51d1-b1e6-46a6-877d-3d43acc1ccac wikiapiary.wikiapiary 3c53fc19-f19f-4c6a-8b23-ab8edb149994 libup-db02.trove 7b1a49d7-2df1-4011-942d-80da14e38f7d buttercup.wikifunctions 4222d13d-aea4-4168-8542-03b5e91f2af8 tools-imagebuilder-2.tools 23e4ccb3-6146-482d-ad4e-1e83e78d1482 deployment-snapshot05.deployment-prep c3be1a8e-94f4-4db8-b3d2-99f74f034ea5 loggerdiscordbot.trove 94f64547-3537-4eae-b7c8-f33f0d5e58bd cloudinfra-internal-puppetserver-1.cloudinfra 32e12f3c-bc0c-4dc1-8ada-abd6e03d1f75 tools-puppetserver-01.tools d277044f-a5db-4939-b1e4-5a142116f9d0 quarry-k8s.trove c4f71bdf-4d88-4f2c-a0bc-d358975a91eb free3.dwl 0126dc3f-02c9-4b07-8550-de103724fdc3 ifis.trove ba0e8a43-352b-48cd-bc86-8f2374fd7447 logging-opensearch-hdd-01.logging 8071f964-80be-41b0-b042-14024965f1ba tools-docker-registry-7.tools c4e76fe0-82c5-4188-8e35-0f63b6d4a216 logging-opensearch-ssd-02.logging ca6525d1-1863-4ab0-b0ef-07c35d063324 taxonbot3.dwl 0038b497-9331-4493-a688-5c85830494e9 tools-docker-registry-8.tools 27c89a15-ee59-4cb2-81f7-845d582ccc88 mwv-puppetserver-01.mediawiki-vagrant a8b2ff57-9841-434b-b233-d8c0064ddb1b mailman-puppetserver-1.mailman fb7fba39-5cef-44ea-a6f3-5b7145f446e8 legacy.wikiapiary fb7fba39-5cef-44ea-a6f3-5b7145f446e8 legacy.wikiapiary b93776fd-b69b-459e-9960-ee69a6b52732 cloudinfra-cloudvps-puppetserver-1.cloudinfra 337b88c3-9009-402d-a7d5-182b1b027fbb copypatrol-dev-db-01.trove eb811cae-2a12-43eb-9d1f-33ee250fe51a copypatrol-prod-db-01.trove c738d3bb-8bd4-4807-8651-78698923385e thistle.trove c02583d4-fa94-4a5c-a4a2-1a50872d45a5 mariadb.trove db1e3259-163c-4a3f-9b5b-a8a37bbb6f5d db01.trove 037795b1-0d21-41e8-840e-ac379426d1e9 osmit-tm4.osmit eb6f591f-69b2-4f16-a7eb-3740b5fd074f db.933ad3ff1e264aada56e6bc3ed9e08f3 50ea5f92-8cb9-4e21-bc65-6fe0b6dd9550 superset-127-jxhvhh7bzlrl-master-0.superset 2faa7a26-c56a-4219-b658-b680212cb64c superset-127-jxhvhh7bzlrl-node-0.superset 70a287ed-3071-4df9-ba1a-ac512cf8749a superset-127-jxhvhh7bzlrl-node-1.superset d182fffb-9f32-41de-96cf-ce5cc5fc95c2 citefix-db.trove fff879b9-8300-4a11-9cf6-3d424be9ffa3 cloudinfra-db04.cloudinfra 47789d5d-4777-455f-8e00-4d24507fe5e7 worker-1.spacemedia e9b6fd4c-d5a3-40dc-968a-ddfcf2222389 worker-2.spacemedia f50f9084-d034-4dae-84fb-80980868335a tools-elastic-4.tools 16ce1807-dd8e-4b87-b4ba-558d2719bbe5 tools-elastic-6.tools 39737a21-257d-4267-9aa6-d2c411501b5f peony-database.globaleducation c8ccdc13-f8b0-4e37-9a2b-795f0bff9f59 videocuttool-bookworm-new.videocuttool c8ccdc13-f8b0-4e37-9a2b-795f0bff9f59 videocuttool-bookworm-new.videocuttool bfad7fbd-53db-4604-aa38-19ffa3e3da02 harbordb.trove 93742dcf-1d18-4c28-8ba0-24ea3219f144 wmcz-stats-test03.wmcz-stats 17c5890b-d783-4f3c-938d-0a3ff2e2e3bc wmcz-stats-wikinside01.wmcz-stats f37f17cf-ee65-4a40-888e-3500120d566d ml-testing.machine-learning 7fd6ada6-2d78-443b-84ce-57aec7cd7a4a ia-upload-prod2.wikisource f75a5008-1b83-4bc2-8cb7-c74f5262d053 pcc-db1002.puppet-diffs d1818008-69c9-4a7b-b642-baeb313df663 patchdemo4-production.catalyst c9bff3a7-f77b-4fea-a31c-b4fb77c4e9e2 deployment-jobrunner05.deployment-prep de63de80-8f02-47e0-9ad2-cfe407ca99be deployment-mediawiki14.deployment-prep 5ff144a5-bb68-44b7-9d0f-39df6b9d3063 deployment-mediawiki13.deployment-prep 93c9a448-d01c-4229-8cf3-29b8a0b765cb deployment-parsoid14.deployment-prep fe1ff5d8-813c-4d06-ae3a-46c05715d305 tools-services-06.tools c99e6edf-89ab-4300-b637-80e11e55438e tools-harbordb.trove 9b5de945-4495-4516-8672-13e525acc557 encoding01.video f9bee5ff-85f5-4cdf-876a-3f4e2e3f3c8e encoding03.video 5a7dcadf-c88e-4d4b-9e44-42e3d636c4ba encoding05.video 85d30118-6397-4bb1-a652-31bb48a1fb2b traffic-puppetserver-bookworm.traffic 5ed094b0-5a2f-42d3-9fe3-a93237351475 encoding06.video fcd555ae-c19c-4b2e-8785-8a9447b263d8 humaniki-prod-bw.wikidumpparse de830403-c93b-4340-9467-d1d8f7ea1628 sample-complex-app-db.trove 3a4d671b-5aef-49ef-b6b4-5cb0c7d00e38 toolsbeta-harbor-2.toolsbeta 421dd73e-8c34-4962-a4e2-61c8997b27d1 quarry-127a-g4ndvpkr5sro-master-0.quarry a1a12a26-55cd-41cd-9696-74773b4db7f0 deployment-wikikube-v127-zwallxbnux67-master-0.deployment-prep 8b99ac12-660d-4267-b841-2d458f8dd9a4 deployment-wikikube-v127-zwallxbnux67-node-0.deployment-prep 65c14b90-8b59-498a-851d-b3c4259fb134 paws-127a-m3mctzr7itba-master-0.paws 3599d6d5-89c2-40ba-95dc-c48c066bd56b paws-127a-m3mctzr7itba-node-2.paws 15f8fdeb-46f9-4853-bdd8-e8c074dce507 paws-127a-m3mctzr7itba-node-3.paws ecac332c-d805-482c-ba71-93838c73b100 paws-127a-m3mctzr7itba-node-1.paws 8373fb59-3b0a-4259-868b-1b4b81c3d10c paws-127a-m3mctzr7itba-node-4.paws caf6628a-f9f7-4f1b-a42a-3cc79b12a758 paws-127a-m3mctzr7itba-node-0.paws caf6628a-f9f7-4f1b-a42a-3cc79b12a758 paws-127a-m3mctzr7itba-node-0.paws ecac332c-d805-482c-ba71-93838c73b100 paws-127a-m3mctzr7itba-node-1.paws [0] https://phabricator.wikimedia.org/T383583 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 2

[Cloud-announce] Share your feedback on the "Toolforge UI" initiative
by Francesco Negri 14 Jan '25

14 Jan '25

I'm cross-posting what was included in today's "Tech News", for people who are not subscribed to that list: "Toolforge tool maintainers can now share their feedback on Toolforge UI, an initiative to provide a web platform that allows creating and managing Toolforge tools through a graphic interface, in addition to existing command-line workflows. This project aims to streamline active maintainers’ tasks, as well as make registration and deployment processes more accessible for new tool creators. The initiative is still at a very early stage, and the Cloud Services team is in the process of collecting feedback from the Toolforge community to help shape the solution to their needs." Read more and share your thoughts about Toolforge UI at this link: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme… -- Francesco Negri (he/him) -- IRC: dhinus Site Reliability Engineer, Cloud Services team Wikimedia Foundation _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

[Cloud-announce] toolforge outage just now
by Andrew Bogott 13 Jan '25

13 Jan '25

Summary: Many toolforge jobs just now failed and/or were interrupted. Everything is now either back online or coming rapidly back online. Explanation: In response to a completely unrelated cloud-vps infra issue (https://phabricator.wikimedia.org/T383583) I needed to migrate many VMs to new hypervisors. I used an updated openstack command for the migration, which turned out to unexpectedly reboot the migrated VMs as part of the migration. Among those VMs rebooted was the toolforge NFS server. This caused the inevitable storm of disconnections, failed mounts, and filesystem lock-ups. This outage was prolonged by the fact that the reboots /also/ temporarily broke puppet which would have otherwise immediately brought the NFS server back online. After staged restarts of various services everything began to recover. As usual, we are now rebooting tools nfs workers in order to reset any locked up nfs file handles. Sorry for the outage! Please follow up here or online if you see any continuing issues. The outage itself is documented as https://phabricator.wikimedia.org/T383625 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 0

LLM services
by Huji Lee 12 Jan '25

12 Jan '25

Hi all, Are there any LLMs available on Cloud services, or are there any plans for them? I think there are many possible use cases. Even free, lightweight LLMs (like LLaMa) could be helpful, e.g. in bots that review edits, categorize pages, etc. Thanks, Huji

6 5

[Cloud-announce] mild dns changes in cloud-vps on Wednesday
by Andrew Bogott 10 Jan '25

10 Jan '25

Tl;dr: Please let me know if you encounter changes in network connectivity between cloud-vps hosts, specifically when routing to floating IPs. Complicated explanation: A previous version of our network setup prevented routing between private and floating IPs within cloud-vps. To work around this we run an agent called 'labs-ip-aliaser' which insertes corresponding private IPs for any dns lookup initiating from within cloud-vps that would otherwise return a floating IP. So, for instance, if you did something like $ ping hostname.wmcloud.org you would actually be pinging vmname.project.eqiad.wikimedia.cloud instead, because that worked and pinging the wmcloud.org address did not. We have since refactored the network so that now reaching public IPs works as expected. So, later in the week I'm going to remove the labs-ip-aliaser hack so that floating-ip-associated hostnames can resolve to the actual floating IPs as expected. This change should be almost entirely invisible to cloud-vps services, but please notify me if you find an interesting edge-case! https://phabricator.wikimedia.org/T374129 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1105443/7 _______________________________________________ Cloud-announce mailing list -- cloud-announce(a)lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.…

1 1

[Mediawiki-api-announce] [Breaking Change] Fix of 'sub' field format in the OAuth identity endpoint
by Gergo Tisza 10 Jan '25

10 Jan '25

The OAuth identity endpoint (the Special:OAuth/identify special page for OAuth 1, the oauth2/resource/profile REST API endpoint for OAuth 2) used to return an incorrectly formatted JSON web token, where value of the 'sub' field (the user's CentralAuth central user ID) was an integer, rather than a string as required by the JWT spec. Due to the latest release of the pyJWT library getting more strict about format validation, this started causing errors in various tools recently. As of this week, this behavior has been fixed for Wikimedia sites, and it has been fixed in all maintained versions (MediaWiki 1.39 and upwards) of the OAuth MediaWiki extension which provides this API. Because the old behavior was a spec violation and caused errors, and it's unlikely the correct behavior would break clients, we are making this fix as a breaking change rather than following the usual API deprecation policies. For more details and discussion see https://phabricator.wikimedia.org/T382139 _______________________________________________ Mediawiki-api-announce mailing list -- mediawiki-api-announce(a)lists.wikimedia.org To unsubscribe send an email to mediawiki-api-announce-leave(a)lists.wikimedia.org

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

Cloud January 2025