-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
There was an outage this morning (mostly of the web server) caused by a fault on the fibre between hemlock and its storage array, which hosts user-store. I have unmounted user-store until the problem is resolved, so www is now working again.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
River Tarnell:
There was an outage this morning (mostly of the web server) caused by a fault on the fibre between hemlock and its storage array, which hosts user-store. I have unmounted user-store until the problem is resolved, so www is now working again.
Hi,
This has been repaired and the storage is now online, so I have remounted user-store everywhere.
The problem seems to have been caused by a faulty HBA port on hemlock. Moving the fibre to the other port resolved the problem, but we will probably need to replace the card.
Our current purchasing plan includes hardware to provide redundant fibre connections to each host, to prevent problems like this in the future, as well as moving user-store to the HA cluster, to protect it from failure of a single host.
- river.
toolserver-l@lists.wikimedia.org