We've had two unexpected system crashes today (cloudvirt1013 and cloudvirt1014) resulting in several VMs being rebooted. I've already contacted those affected by the cloudvirt1013 crash directly. The instances rebooted as part of the cloudvirt1014 are listed below.
For the moment we're going to leave 1014 intact and hope that this was a one-time failure. The issue will be tracked here: https://phabricator.wikimedia.org/T241492
Sorry for the downtime! Here is the list of affected VMs:
| 367cec0b-a95a-4c64-a9d0-2c88553b3a3e | jeh-cephmon03 | | 763ca86d-edea-40ba-8c8a-d128cdb8e5ed | filippo-log-buster01 | | 55831fc4-7a34-4d61-a9fb-f2d060b5bd7b | staging | | f0c8cf87-e0be-4b77-a5d4-535782bdf05e | opusmt | | 528eda19-d5f3-4ceb-a75f-81d03dc2be8a | mw01 | | 4e7b37bf-2037-4179-9b03-63ea0f032345 | grantreview-04 | | 5f96add9-ccc8-4308-955d-ec1ce4b32cc8 | cloud-puppetmaster-03 | | 155ea43c-4850-4c44-8462-9b8e291a0b16 | commtech-wikiwho | | 95f23fb7-3b84-4f2b-9802-11b385247c1e | toolsbeta-test-k8s-haproxy-2 | | b904683b-4c94-4d9d-bafd-a17af4ee42d3 | debmonitor-pm | | 68065e80-8b4d-461b-bd34-f027d079330b | toolsbeta-test-k8s-control-1 | | 59922260-bd39-4bdc-b233-feb78ddb2665 | toolsbeta-test-k8s-etcd-2 | | 7637ae11-247d-47ad-aa6d-f7e1b31c836e | toolsbeta-test-k8s-etcd-1 | | 7ed021f3-ee41-4bd9-bf3a-bd43ce2aaa59 | jmm-debm-01 | | 3715977d-d619-406e-8220-24aef7f077b7 | archiva | | 594b4b48-0bbb-4296-bb52-ef7861036ef6 | xtools-dev05 | | 817e4922-bd49-4a07-8476-ab4e544420ce | ores-web-05 | | c3454178-e377-4740-8317-916644d29746 | roebling | | e5840383-05ea-4721-9c0b-d198d1f3b0ad | ores-web-04 | | a044ce46-fe45-4343-8ef5-2ea54408036d | canary1014-01 | | fc5e2197-5de3-4ff2-8ae3-9656a6398c2d | cloudinfra-db02 | | 64d0dc6a-058e-48ef-bcd6-d5b53e678c18 | Krypton | | 22892991-bd1f-402c-a3d1-16aca90bd721 | discovery-production-02 | | 2e2d0b7f-a175-4102-9cef-1b57e19f3156 | tools-sgegrid-shadow | | ce430b95-cf87-4e5f-a9fe-4f4018945665 | maps-tiles1 | | 0ab42fcf-f405-4159-90b9-e61a973f3bd3 | tools-sgecron-01 | | cbf687b4-f6aa-4202-a3f4-ff6ce59793a4 | wikitextexp-base-1002 | | 070e5515-f411-435e-aafc-66a068a327fb | clm-web-01 | | 37309f7f-07b8-45bc-a017-574b500e6ebc | accounts-appserver4 | | d8062efc-59df-42ee-bed0-011cb6a14b16 | tofawiki02 | | 770397da-6555-46e1-b79c-7f6321ee02e8 | packagist-mirror1 | | 6f0423b6-b3b9-44f3-b7e8-d51dfa9c6744 | deployment-elastic06 | | ca3c5b5b-1358-41a3-accd-3ba18c658542 | deployment-changeprop | | 63ae9220-64ec-440c-b46b-52c8fc1c44c2 | deployment-restbase02 | | d0220309-5972-4710-a617-d57532a47a05 | deployment-imagescaler01 | | f04b0bdd-852d-4ff5-a2a0-4006e064c51b | deployment-kafka-jumbo-1 | | 06c8f1f1-2070-4d05-b97b-386c6bf5636b | deployment-memc07 | | 09b3a1c6-4a8d-43a4-9730-c4a2e62229d9 | deployment-eventlog05 | | 234a724d-68bd-4f72-9e88-185246aba3c4 | deployment-cpjobqueue | | 4bff32b7-316a-40d4-9de4-3a13a7ebf62a | deployment-mediawiki-07 | | 25dedc51-1e89-407e-b57c-8af7fc483bc8 | deployment-chromium01 | | a85b249d-48c6-4b8d-b563-656f5bd157d2 | deployment-puppetdb02 | | 499e6f92-87b8-4e6d-9982-237c120eaa1e | deployment-puppetmaster03 | | eed81e86-2874-4740-a8c2-fee29ced046d | deployment-cache-text05 | | 7eaa6920-558c-48ba-8711-d0332d0408e9 | whgi | | 8af61b76-e559-4572-a049-d6b550e3f7ef | wikilabels | | 9cc4397d-5efb-40fa-9282-9338afc19453 | gitservices | | 7bfb8119-7d9a-4abe-8647-50879a023a3f | wikilabels-02 | | d94c37a2-b71c-4ea4-91ae-a3a236166392 | af-puppetdb02 | | 1aac396f-9bd8-4eb8-8edd-6e0d0bdef2ed | missing-sections | | c4a27ce0-5a56-4bf1-b914-8a158fd52e72 | ores-lb-03 | | deb637c2-cfce-4bcc-aa10-49f0f8241bdd | matrix-synapse-01 | | 2dee30c9-de3a-425f-b8d0-eec5e7a20585 | captcha-tf-43 | | fdb7710a-de83-40c5-beae-98c0ab8ea7a8 | k4-2 |