We lost a KVM host at around 7:20 UTC. Because we use local storage for
instances there are a number of them that are down. Toolforge suffered a
few losses but it seems to have been few enough that GridEngine and
Kubernetes users are unaffected at this time . The initial task is T187292
(with a list of instances), and an incident report will follow. We hope to
recover all of the instances that are down but it will take time to sort
through.
--
Chase Pettet
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC