<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Toolslabs will be back soon? When approximately?<br><br><div>> Date: Wed, 17 Jun 2015 23:09:11 -0400<br>> From: marc@uberbox.org<br>> To: abogott@wikimedia.org; labs-l@lists.wikimedia.org<br>> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]<br>> <br>> On 15-06-17 09:29 PM, Andrew Bogott wrote:<br>> > Coren is rebooting and fscking the system -- with luck it'll be up<br>> > again within the hour.<br>> <br>> Fortune does not smile upon us; the fsck is taking a LONG time to<br>> progress (over a 40T filesystem), but the repair goes apace.<br>> <br>> There is already indication of some lost data - though the failures are<br>> likely to date from a past crash and may have gone unnoticed for some time.<br>> <br>> Either way, we have a fairly recent (8-9 days) backup of the entire<br>> filesystem so we may be able to recover lost or corrupted files if any<br>> turn up.<br>> <br>> Worst case scenario is a catastrophic loss of the entire filesystem<br>> (which does not seem likely at this time), in which case we will restore<br>> the filesystem completely from the backup.<br>> <br>> The repair process will take at least a few more hours, during which I<br>> will be keeping watch. I will update this list as soon as I have a<br>> better assessment of the damage, if any.<br>> <br>> -- Marc<br>> <br>> <br>> _______________________________________________<br>> Labs-l mailing list<br>> Labs-l@lists.wikimedia.org<br>> https://lists.wikimedia.org/mailman/listinfo/labs-l<br></div> </div></body>
</html>