-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
One of the database servers for these clusters (cassia) has a disk failure of some kind. All databases should be accessible now, but user databases will be missing until the problem is resolved.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
This seems to have been some kind of controller problem. After the power cycle, the server looks fine, and user databases should be available again. The server might be a bit slow since the RAID array is rebuilding.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Unfortunately this problem came back. I will set cassia to read-only, and move user databases to the secondary server (hyacinth) until the problem is fixed. This will cause an outage of user databases for a few hours. Since we're now using separate server instances for each cluster, databases will be imported as describe in [0].
- river.
[0] http://lists.wikimedia.org/pipermail/toolserver-announce/2011-June/000440.ht...
toolserver-l@lists.wikimedia.org