We've had two unexpected system crashes today (cloudvirt1013 and
cloudvirt1014) resulting in several VMs being rebooted. I've already
contacted those affected by the cloudvirt1013 crash directly. The
instances rebooted as part of the cloudvirt1014 are listed below.
For the moment we're going to leave 1014 intact and hope that this was a
one-time failure. The issue will be tracked here:
https://phabricator.wikimedia.org/T241492
Sorry for the downtime! Here is the list of affected VMs:
| 367cec0b-a95a-4c64-a9d0-2c88553b3a3e | jeh-cephmon03 |
| 763ca86d-edea-40ba-8c8a-d128cdb8e5ed | filippo-log-buster01 |
| 55831fc4-7a34-4d61-a9fb-f2d060b5bd7b | staging |
| f0c8cf87-e0be-4b77-a5d4-535782bdf05e | opusmt |
| 528eda19-d5f3-4ceb-a75f-81d03dc2be8a | mw01 |
| 4e7b37bf-2037-4179-9b03-63ea0f032345 | grantreview-04 |
| 5f96add9-ccc8-4308-955d-ec1ce4b32cc8 | cloud-puppetmaster-03 |
| 155ea43c-4850-4c44-8462-9b8e291a0b16 | commtech-wikiwho |
| 95f23fb7-3b84-4f2b-9802-11b385247c1e | toolsbeta-test-k8s-haproxy-2 |
| b904683b-4c94-4d9d-bafd-a17af4ee42d3 | debmonitor-pm |
| 68065e80-8b4d-461b-bd34-f027d079330b | toolsbeta-test-k8s-control-1 |
| 59922260-bd39-4bdc-b233-feb78ddb2665 | toolsbeta-test-k8s-etcd-2 |
| 7637ae11-247d-47ad-aa6d-f7e1b31c836e | toolsbeta-test-k8s-etcd-1 |
| 7ed021f3-ee41-4bd9-bf3a-bd43ce2aaa59 | jmm-debm-01 |
| 3715977d-d619-406e-8220-24aef7f077b7 | archiva |
| 594b4b48-0bbb-4296-bb52-ef7861036ef6 | xtools-dev05 |
| 817e4922-bd49-4a07-8476-ab4e544420ce | ores-web-05 |
| c3454178-e377-4740-8317-916644d29746 | roebling |
| e5840383-05ea-4721-9c0b-d198d1f3b0ad | ores-web-04 |
| a044ce46-fe45-4343-8ef5-2ea54408036d | canary1014-01 |
| fc5e2197-5de3-4ff2-8ae3-9656a6398c2d | cloudinfra-db02 |
| 64d0dc6a-058e-48ef-bcd6-d5b53e678c18 | Krypton |
| 22892991-bd1f-402c-a3d1-16aca90bd721 | discovery-production-02 |
| 2e2d0b7f-a175-4102-9cef-1b57e19f3156 | tools-sgegrid-shadow |
| ce430b95-cf87-4e5f-a9fe-4f4018945665 | maps-tiles1 |
| 0ab42fcf-f405-4159-90b9-e61a973f3bd3 | tools-sgecron-01 |
| cbf687b4-f6aa-4202-a3f4-ff6ce59793a4 | wikitextexp-base-1002 |
| 070e5515-f411-435e-aafc-66a068a327fb | clm-web-01 |
| 37309f7f-07b8-45bc-a017-574b500e6ebc | accounts-appserver4 |
| d8062efc-59df-42ee-bed0-011cb6a14b16 | tofawiki02 |
| 770397da-6555-46e1-b79c-7f6321ee02e8 | packagist-mirror1 |
| 6f0423b6-b3b9-44f3-b7e8-d51dfa9c6744 | deployment-elastic06 |
| ca3c5b5b-1358-41a3-accd-3ba18c658542 | deployment-changeprop |
| 63ae9220-64ec-440c-b46b-52c8fc1c44c2 | deployment-restbase02 |
| d0220309-5972-4710-a617-d57532a47a05 | deployment-imagescaler01 |
| f04b0bdd-852d-4ff5-a2a0-4006e064c51b | deployment-kafka-jumbo-1 |
| 06c8f1f1-2070-4d05-b97b-386c6bf5636b | deployment-memc07 |
| 09b3a1c6-4a8d-43a4-9730-c4a2e62229d9 | deployment-eventlog05 |
| 234a724d-68bd-4f72-9e88-185246aba3c4 | deployment-cpjobqueue |
| 4bff32b7-316a-40d4-9de4-3a13a7ebf62a | deployment-mediawiki-07 |
| 25dedc51-1e89-407e-b57c-8af7fc483bc8 | deployment-chromium01 |
| a85b249d-48c6-4b8d-b563-656f5bd157d2 | deployment-puppetdb02 |
| 499e6f92-87b8-4e6d-9982-237c120eaa1e | deployment-puppetmaster03 |
| eed81e86-2874-4740-a8c2-fee29ced046d | deployment-cache-text05 |
| 7eaa6920-558c-48ba-8711-d0332d0408e9 | whgi |
| 8af61b76-e559-4572-a049-d6b550e3f7ef | wikilabels |
| 9cc4397d-5efb-40fa-9282-9338afc19453 | gitservices |
| 7bfb8119-7d9a-4abe-8647-50879a023a3f | wikilabels-02 |
| d94c37a2-b71c-4ea4-91ae-a3a236166392 | af-puppetdb02 |
| 1aac396f-9bd8-4eb8-8edd-6e0d0bdef2ed | missing-sections |
| c4a27ce0-5a56-4bf1-b914-8a158fd52e72 | ores-lb-03 |
| deb637c2-cfce-4bcc-aa10-49f0f8241bdd | matrix-synapse-01 |
| 2dee30c9-de3a-425f-b8d0-eec5e7a20585 | captcha-tf-43 |
| fdb7710a-de83-40c5-beae-98c0ab8ea7a8 | k4-2 |
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce