hello,
some extended maintenance is currently taking place on the NFS server for /home, so hemlock is offline. this shouldn't take more than a few hours. i'll post a full explanation when it's finished.
- river.
the maintenance is finished now. the problem was caused by filesystem corruption on clematis:/aux0, the QFS filesystem where hemlock's /home is currently mounted from, when the connection to the iSCSI array was broken. i have replaced this filesystem with a VxFS filesystem, which should be more resiliant against problems like this.
the iSCSI problem was my fault, so sorry for that. to prevent it happening in the future, i've asked for the array to be connected directly to clematis's NIC, which should be more reliable. in the longer term the plan is to move hemlock's /home back to a local array; this should happen either this month or next, when a new array is installed at knams.
a very small number of files were unrecoverable from the damaged /home. if any of your files are missing, mail ts-admins or file a bug and they can be restored from a backup.
- river.
toolserver-l@lists.wikimedia.org