-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
hi,
the Toolserver is currently offline due to an issue with the NFS server. we are working on the issue and have a ticket open with Sun, but at this point there is no ETA.
if it looks like the issue will take some time to resolve, we will restore the last /home backup onto another server and use that. however, that means losing about 12 hours worth of changes, so we will try to avoid it if possible.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
River Tarnell:
the Toolserver is currently offline due to an issue with the NFS server. we are working on the issue and have a ticket open with Sun, but at this point there is no ETA.
update: hyacinth is now online without /home. this means stable is up, but the rest of the Toolserver is still offline.
at this point we understand the issue but not the cause. i have two separate plans to restore service; preferably by recovering /home, but if that is not possible within the next few hours, i will rebuild the server and restore from the last backup, dated 2009-08-24 at 5:00AM UTC.
once service is restored i will provide a post-mortem report on the issue.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
after further investigation we have determined that the best course of action is to restore from a backup. i have imported the last /home backup (2009-08-24 05:00 UTC) to hemlock, and temporarily mounted this as /home, so the Toolserver is now up.
i will rebuild the NFS server now (which will take a couple of hours at least) and then start migrating data back to it. there will be some further downtime while this is done.
i will be making some changes to reduce issues like this in the future, which i will document later.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
hi,
i've finished repairing hyacinth (the NFS server) and i'm about to start copying /home back. this will take an hour or two during which the Toolserver will be offline again. assuming there are no problems, this should be the last downtime.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
hi,
the maintenance is now mostly finished and the Toolserver is back up. i'm now working on bringing stable back.
- river.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
hi,
maintenance on stable is now finished and it should be up. the system was reinstalled, so if you notice anything odd (like missing configuration or software) please let me know.
- river.
toolserver-announce@lists.wikimedia.org