-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
On the morning (UTC) of March 1 we will install some new hardware at the Toolserver. Services will be affected as follows:
Service | Expected impact ----------------------------+-------------------------------- Databases: s3, s4, s6, s7 | User databases unavailable for | < 10 minutes. user-store filesystem | Unavailable for < 10 minutes.
Start time: Tuesday, 1st March, 1PM UTC (estimated) http://time.tcx.org.uk/utc/2011/3/1/13/0
End time: Tuesday, 1st March, 4PM UTC (estimated) http://time.tcx.org.uk/utc/2011/3/1/16/0
Details:
We will install new storage for databases and the user-store filesystem. This requires installing HBAs in three servers: hemlock, cassia, and hyacinth, which requires downtime. We will try to stagger the installation of the two database servers so that those clusters remaining accessible during the maintenance, but user databases will be unavailable while hyacinth is down.
Before the maintenance on hemlock, we will forcibly unmount the user-store filesystem. Any program accessing it at the time will receive an I/O error.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
River Tarnell:
On the morning (UTC) of March 1 we will install some new hardware at the Toolserver. Services will be affected as follows:
Service | Expected impact ----------------------------+-------------------------------- Databases: s3, s4, s6, s7 | User databases unavailable for | < 10 minutes. user-store filesystem | Unavailable for < 10 minutes.
This maintenance has been postponed until tomorrow due to the unexpectedly long time it took to rack the new hardware.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The database side of this is finished. However, due to some problem with hemlock's disk array, the user-store filesystem is currently being fscked; this might take a few hours. user-store will be unavailable until then.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
River Tarnell:
user-store will be unavailable until then.
user-store is now available again and the maintenance is complete.
- river.
toolserver-announce@lists.wikimedia.org