[Labs-l] NFS outage in progress [UPDATE]

[Marc A. Pelletier]

On 15-06-17 09:29 PM, Andrew Bogott wrote:
> Coren is rebooting and fscking the system -- with luck it'll be up
> again within the hour.

Fortune does not smile upon us; the fsck is taking a LONG time to
progress (over a 40T filesystem), but the repair goes apace.

There is already indication of some lost data - though the failures are
likely to date from a past crash and may have gone unnoticed for some time.

Either way, we have a fairly recent (8-9 days) backup of the entire
filesystem so we may be able to recover lost or corrupted files if any
turn up.

Worst case scenario is a catastrophic loss of the entire filesystem
(which does not seem likely at this time), in which case we will restore
the filesystem completely from the backup.

The repair process will take at least a few more hours, during which I
will be keeping watch.  I will update this list as soon as I have a
better assessment of the damage, if any.

-- Marc