-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

hi,

today we experienced a 6-second power failure at SARA, the colocation facility where the toolserver is hosted. this affected both us and Wikimedia (who provide our hosting there).

this affected all servers except willow (stable), which was connected to a different circuit and remained up. while hemlock also didn't lose power, its NFS server (zedler) and LDAP server (nightshade) were both down.

as no administrator was immediately available, and the platform did not restart graciously, total downtime was around 5 hours.

i will investigate the problems preventing an automatic recovery after restart, to reduce the downtime if this should happen again.

this downtime does not affect the maintenance currently scheduled for Monday.

- river.