[Toolserver-l] downtime today (04 Oct): zedler, yarrow, vandale, nightshade, hemlock

[River Tarnell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

hi,

today we experienced a 6-second power failure at SARA, the colocation facility
where the toolserver is hosted.  this affected both us and Wikimedia (who
provide our hosting there).

this affected all servers except willow (stable), which was connected to a
different circuit and remained up.  while hemlock also didn't lose power, its
NFS server (zedler) and LDAP server (nightshade) were both down.

as no administrator was immediately available, and the platform did not restart
graciously, total downtime was around 5 hours.  

i will investigate the problems preventing an automatic recovery after restart,
to reduce the downtime if this should happen again.

this downtime does not affect the maintenance currently scheduled for Monday.

	- river.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (SunOS)

iEYEARECAAYFAkjnzMkACgkQIXd7fCuc5vLDbwCdHVVWqxZGrLZSQ6VmLQiUU99P
frwAoI1/wQYiTniuO0HqMtv5i97Im/SG
=O3Eh
-----END PGP SIGNATURE-----