We've had two unexpected system crashes today (cloudvirt1013 and cloudvirt1014) resulting in several VMs being rebooted. I've already contacted those affected by the cloudvirt1013 crash directly. The instances rebooted as part of the cloudvirt1014 are listed below. For the moment we're going to leave 1014 intact and hope that this was a one-time failure. The issue will be tracked here: https://phabricator.wikimedia.org/T241492 Sorry for the downtime! Here is the list of affected VMs: | 367cec0b-a95a-4c64-a9d0-2c88553b3a3e | jeh-cephmon03 | | 763ca86d-edea-40ba-8c8a-d128cdb8e5ed | filippo-log-buster01 | | 55831fc4-7a34-4d61-a9fb-f2d060b5bd7b | staging | | f0c8cf87-e0be-4b77-a5d4-535782bdf05e | opusmt | | 528eda19-d5f3-4ceb-a75f-81d03dc2be8a | mw01 | | 4e7b37bf-2037-4179-9b03-63ea0f032345 | grantreview-04 | | 5f96add9-ccc8-4308-955d-ec1ce4b32cc8 | cloud-puppetmaster-03 | | 155ea43c-4850-4c44-8462-9b8e291a0b16 | commtech-wikiwho | | 95f23fb7-3b84-4f2b-9802-11b385247c1e | toolsbeta-test-k8s-haproxy-2 | | b904683b-4c94-4d9d-bafd-a17af4ee42d3 | debmonitor-pm | | 68065e80-8b4d-461b-bd34-f027d079330b | toolsbeta-test-k8s-control-1 | | 59922260-bd39-4bdc-b233-feb78ddb2665 | toolsbeta-test-k8s-etcd-2 | | 7637ae11-247d-47ad-aa6d-f7e1b31c836e | toolsbeta-test-k8s-etcd-1 | | 7ed021f3-ee41-4bd9-bf3a-bd43ce2aaa59 | jmm-debm-01 | | 3715977d-d619-406e-8220-24aef7f077b7 | archiva | | 594b4b48-0bbb-4296-bb52-ef7861036ef6 | xtools-dev05 | | 817e4922-bd49-4a07-8476-ab4e544420ce | ores-web-05 | | c3454178-e377-4740-8317-916644d29746 | roebling | | e5840383-05ea-4721-9c0b-d198d1f3b0ad | ores-web-04 | | a044ce46-fe45-4343-8ef5-2ea54408036d | canary1014-01 | | fc5e2197-5de3-4ff2-8ae3-9656a6398c2d | cloudinfra-db02 | | 64d0dc6a-058e-48ef-bcd6-d5b53e678c18 | Krypton | | 22892991-bd1f-402c-a3d1-16aca90bd721 | discovery-production-02 | | 2e2d0b7f-a175-4102-9cef-1b57e19f3156 | tools-sgegrid-shadow | | ce430b95-cf87-4e5f-a9fe-4f4018945665 | maps-tiles1 | | 0ab42fcf-f405-4159-90b9-e61a973f3bd3 | tools-sgecron-01 | | cbf687b4-f6aa-4202-a3f4-ff6ce59793a4 | wikitextexp-base-1002 | | 070e5515-f411-435e-aafc-66a068a327fb | clm-web-01 | | 37309f7f-07b8-45bc-a017-574b500e6ebc | accounts-appserver4 | | d8062efc-59df-42ee-bed0-011cb6a14b16 | tofawiki02 | | 770397da-6555-46e1-b79c-7f6321ee02e8 | packagist-mirror1 | | 6f0423b6-b3b9-44f3-b7e8-d51dfa9c6744 | deployment-elastic06 | | ca3c5b5b-1358-41a3-accd-3ba18c658542 | deployment-changeprop | | 63ae9220-64ec-440c-b46b-52c8fc1c44c2 | deployment-restbase02 | | d0220309-5972-4710-a617-d57532a47a05 | deployment-imagescaler01 | | f04b0bdd-852d-4ff5-a2a0-4006e064c51b | deployment-kafka-jumbo-1 | | 06c8f1f1-2070-4d05-b97b-386c6bf5636b | deployment-memc07 | | 09b3a1c6-4a8d-43a4-9730-c4a2e62229d9 | deployment-eventlog05 | | 234a724d-68bd-4f72-9e88-185246aba3c4 | deployment-cpjobqueue | | 4bff32b7-316a-40d4-9de4-3a13a7ebf62a | deployment-mediawiki-07 | | 25dedc51-1e89-407e-b57c-8af7fc483bc8 | deployment-chromium01 | | a85b249d-48c6-4b8d-b563-656f5bd157d2 | deployment-puppetdb02 | | 499e6f92-87b8-4e6d-9982-237c120eaa1e | deployment-puppetmaster03 | | eed81e86-2874-4740-a8c2-fee29ced046d | deployment-cache-text05 | | 7eaa6920-558c-48ba-8711-d0332d0408e9 | whgi | | 8af61b76-e559-4572-a049-d6b550e3f7ef | wikilabels | | 9cc4397d-5efb-40fa-9282-9338afc19453 | gitservices | | 7bfb8119-7d9a-4abe-8647-50879a023a3f | wikilabels-02 | | d94c37a2-b71c-4ea4-91ae-a3a236166392 | af-puppetdb02 | | 1aac396f-9bd8-4eb8-8edd-6e0d0bdef2ed | missing-sections | | c4a27ce0-5a56-4bf1-b914-8a158fd52e72 | ores-lb-03 | | deb637c2-cfce-4bcc-aa10-49f0f8241bdd | matrix-synapse-01 | | 2dee30c9-de3a-425f-b8d0-eec5e7a20585 | captcha-tf-43 | | fdb7710a-de83-40c5-beae-98c0ab8ea7a8 | k4-2 | _______________________________________________ Wikimedia Cloud Services announce mailing list Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud-announce _______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud

I doubt it's that as the tools project didn't loose any exec instances in this issue that I'm aware of. Anyway I started that tool up krenair@tools-sgebastion-07:~$ sudo become parliamentdiagram tools.parliamentdiagram@tools-sgebastion-07:~$ webservice status Your webservice is not running tools.parliamentdiagram@tools-sgebastion-07:~$ webservice start Starting webservice... tools.parliamentdiagram@tools-sgebastion-07:~$ webservice status Your webservice of type lighttpd is running On Sat, 28 Dec 2019 at 11:59, David Richfield <davidrichfield(a)gmail.com> wrote: > Hi! > > The parliament diagram tool ( > https://tools.wmflabs.org/parliamentdiagram/parlitest.php) is down, > and I'm on holiday away from my computers. Is this due to this issue, and > what should I be doing about it? > > Thanks > > David > > Andrew Bogott <abogott(a)wikimedia.org> schrieb am Fr., 27. Dez. 2019, > 18:51: > >> We've had two unexpected system crashes today (cloudvirt1013 and >> cloudvirt1014) resulting in several VMs being rebooted. I've already >> contacted those affected by the cloudvirt1013 crash directly. The >> instances rebooted as part of the cloudvirt1014 are listed below. >> >> For the moment we're going to leave 1014 intact and hope that this >> was a >> one-time failure. The issue will be tracked here: >> https://phabricator.wikimedia.org/T241492 >> >> Sorry for the downtime! Here is the list of affected VMs: >> >> >> | 367cec0b-a95a-4c64-a9d0-2c88553b3a3e | jeh-cephmon03 >> | >> | 763ca86d-edea-40ba-8c8a-d128cdb8e5ed | filippo-log-buster01 >> | >> | 55831fc4-7a34-4d61-a9fb-f2d060b5bd7b | staging >> | >> | f0c8cf87-e0be-4b77-a5d4-535782bdf05e | opusmt >> | >> | 528eda19-d5f3-4ceb-a75f-81d03dc2be8a | mw01 >> | >> | 4e7b37bf-2037-4179-9b03-63ea0f032345 | grantreview-04 >> | >> | 5f96add9-ccc8-4308-955d-ec1ce4b32cc8 | cloud-puppetmaster-03 >> | >> | 155ea43c-4850-4c44-8462-9b8e291a0b16 | commtech-wikiwho >> | >> | 95f23fb7-3b84-4f2b-9802-11b385247c1e | toolsbeta-test-k8s-haproxy-2 >> | >> | b904683b-4c94-4d9d-bafd-a17af4ee42d3 | debmonitor-pm >> | >> | 68065e80-8b4d-461b-bd34-f027d079330b | toolsbeta-test-k8s-control-1 >> | >> | 59922260-bd39-4bdc-b233-feb78ddb2665 | toolsbeta-test-k8s-etcd-2 >> | >> | 7637ae11-247d-47ad-aa6d-f7e1b31c836e | toolsbeta-test-k8s-etcd-1 >> | >> | 7ed021f3-ee41-4bd9-bf3a-bd43ce2aaa59 | jmm-debm-01 >> | >> | 3715977d-d619-406e-8220-24aef7f077b7 | archiva >> | >> | 594b4b48-0bbb-4296-bb52-ef7861036ef6 | xtools-dev05 >> | >> | 817e4922-bd49-4a07-8476-ab4e544420ce | ores-web-05 >> | >> | c3454178-e377-4740-8317-916644d29746 | roebling >> | >> | e5840383-05ea-4721-9c0b-d198d1f3b0ad | ores-web-04 >> | >> | a044ce46-fe45-4343-8ef5-2ea54408036d | canary1014-01 >> | >> | fc5e2197-5de3-4ff2-8ae3-9656a6398c2d | cloudinfra-db02 >> | >> | 64d0dc6a-058e-48ef-bcd6-d5b53e678c18 | Krypton >> | >> | 22892991-bd1f-402c-a3d1-16aca90bd721 | discovery-production-02 >> | >> | 2e2d0b7f-a175-4102-9cef-1b57e19f3156 | tools-sgegrid-shadow >> | >> | ce430b95-cf87-4e5f-a9fe-4f4018945665 | maps-tiles1 >> | >> | 0ab42fcf-f405-4159-90b9-e61a973f3bd3 | tools-sgecron-01 >> | >> | cbf687b4-f6aa-4202-a3f4-ff6ce59793a4 | wikitextexp-base-1002 >> | >> | 070e5515-f411-435e-aafc-66a068a327fb | clm-web-01 >> | >> | 37309f7f-07b8-45bc-a017-574b500e6ebc | accounts-appserver4 >> | >> | d8062efc-59df-42ee-bed0-011cb6a14b16 | tofawiki02 >> | >> | 770397da-6555-46e1-b79c-7f6321ee02e8 | packagist-mirror1 >> | >> | 6f0423b6-b3b9-44f3-b7e8-d51dfa9c6744 | deployment-elastic06 >> | >> | ca3c5b5b-1358-41a3-accd-3ba18c658542 | deployment-changeprop >> | >> | 63ae9220-64ec-440c-b46b-52c8fc1c44c2 | deployment-restbase02 >> | >> | d0220309-5972-4710-a617-d57532a47a05 | deployment-imagescaler01 >> | >> | f04b0bdd-852d-4ff5-a2a0-4006e064c51b | deployment-kafka-jumbo-1 >> | >> | 06c8f1f1-2070-4d05-b97b-386c6bf5636b | deployment-memc07 >> | >> | 09b3a1c6-4a8d-43a4-9730-c4a2e62229d9 | deployment-eventlog05 >> | >> | 234a724d-68bd-4f72-9e88-185246aba3c4 | deployment-cpjobqueue >> | >> | 4bff32b7-316a-40d4-9de4-3a13a7ebf62a | deployment-mediawiki-07 >> | >> | 25dedc51-1e89-407e-b57c-8af7fc483bc8 | deployment-chromium01 >> | >> | a85b249d-48c6-4b8d-b563-656f5bd157d2 | deployment-puppetdb02 >> | >> | 499e6f92-87b8-4e6d-9982-237c120eaa1e | deployment-puppetmaster03 >> | >> | eed81e86-2874-4740-a8c2-fee29ced046d | deployment-cache-text05 >> | >> | 7eaa6920-558c-48ba-8711-d0332d0408e9 | whgi >> | >> | 8af61b76-e559-4572-a049-d6b550e3f7ef | wikilabels >> | >> | 9cc4397d-5efb-40fa-9282-9338afc19453 | gitservices >> | >> | 7bfb8119-7d9a-4abe-8647-50879a023a3f | wikilabels-02 >> | >> | d94c37a2-b71c-4ea4-91ae-a3a236166392 | af-puppetdb02 >> | >> | 1aac396f-9bd8-4eb8-8edd-6e0d0bdef2ed | missing-sections >> | >> | c4a27ce0-5a56-4bf1-b914-8a158fd52e72 | ores-lb-03 >> | >> | deb637c2-cfce-4bcc-aa10-49f0f8241bdd | matrix-synapse-01 >> | >> | 2dee30c9-de3a-425f-b8d0-eec5e7a20585 | captcha-tf-43 >> | >> | fdb7710a-de83-40c5-beae-98c0ab8ea7a8 | k4-2 >> | >> >> >> _______________________________________________ >> Wikimedia Cloud Services announce mailing list >> Cloud-announce(a)lists.wikimedia.org (formerly >> labs-announce(a)lists.wikimedia.org) >> https://lists.wikimedia.org/mailman/listinfo/cloud-announce >> _______________________________________________ >> Wikimedia Cloud Services mailing list >> Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) >> https://lists.wikimedia.org/mailman/listinfo/cloud > > _______________________________________________ > Wikimedia Cloud Services mailing list > Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) > https://lists.wikimedia.org/mailman/listinfo/cloud _______________________________________________ Wikimedia Cloud Services mailing list Cloud(a)lists.wikimedia.org (formerly labs-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud