cloudvirt1004 is one of our oldest generation of hypervisor servers. The hypervisor servers are the machines which actually run the virtual machine instances for Cloud VPS projects. This physical host is experiencing an active hard disk and/or RAID controller failure. The Cloud Services team is actively attempting to fix the server and evacuate all instances running on it to other hypervisors.
See https://phabricator.wikimedia.org/T250869 for more information and progress updates.
The following projects and instances are affected:
* cloudvirt-canary ** canary1004-01.cloudvirt-canary.eqiad.wmflabs
* commonsarchive ** commonsarchive-mwtest.commonsarchive.eqiad.wmflabs
* deployment-prep ** deployment-echostore01.deployment-prep.eqiad.wmflabs ** deployment-schema-2.deployment-prep.eqiad.wmflabs
* incubator ** incubator-mw.incubator.eqiad.wmflabs
* machine-vision ** visionoid.machine-vision.eqiad.wmflabs
* ogvjs-integration ** media-streaming.ogvjs-integration.eqiad.wmflabs
* services ** Esther-outreachy-intern.services.eqiad.wmflabs
* shiny-r ** discovery-testing-02.shiny-r.eqiad.wmflabs
* tools ** tools-k8s-worker-38.tools.eqiad.wmflabs ** tools-k8s-worker-52.tools.eqiad.wmflabs ** tools-sgeexec-0901.tools.eqiad.wmflabs ** tools-sgewebgrid-lighttpd-0918.tools.eqiad.wmflabs ** tools-sgewebgrid-lighttpd-0919.tools.eqiad.wmflabs
* toolsbeta ** toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad.wmflabs
* wikidata-autodesc ** wikidata-autodesc.wikidata-autodesc.eqiad.wmflabs
* wikilink ** wikilink-prod.wikilink.eqiad.wmflabs
Bryan, on behalf of the Cloud VPS admins and Cloud Services team