[Labs-l] NFS outage in progress [UPDATE]

[Petr Bena]

Kdyz tady bude kazdy mluvit svym jazykem tak si moc nepokecame...

To be fair, labs do work - most of the time, and afaik it doesn't have
so big budget, so blaming the foundation is not very nice. It was
always meant for development. Ops are actually working hard on it to
be as much stable as possible.


On Thu, Jun 18, 2015 at 3:49 PM, Maximilian Doerr
<maximilian.doerr at gmail.com> wrote:
> Any ETA when I can SSH back in and restore my bots?
>
> Cyberpower678
> English Wikipedia Account Creation Team
> Mailing List Moderator
>
>
>
> On Jun 18, 2015, at 09:48, Steinsplitter Wiki <steinsplitter-wiki at live.com>
> wrote:
>
> the foundation isn't able to provide a service that works in real life.
>
> ________________________________
> From: maximilian.doerr at gmail.com
> Date: Thu, 18 Jun 2015 09:47:07 -0400
> To: labs-l at lists.wikimedia.org
> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>
> Mist.  Ich bekomme wieder beschwerte dass meine Bots schon wieder nicht
> funktionieren. :/
>
> Cyberpower678
> English Wikipedia Account Creation Team
> Mailing List Moderator
>
>
>
> On Jun 18, 2015, at 09:45, Steinsplitter Wiki <steinsplitter-wiki at live.com>
> wrote:
>
>>> Hat was hier los ist damit was zum tun?
>
> ja :(
>
> ________________________________
> From: maximilian.doerr at gmail.com
> Date: Thu, 18 Jun 2015 09:41:53 -0400
> To: labs-l at lists.wikimedia.org
> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>
> Ich kann weder einloggen oder meine Bot Seiten aufrufen.  Hat was hier los
> ist damit was zum tun?  ACC funktioniert auch nicht.
>
> Cyberpower678
> English Wikipedia Account Creation Team
> Mailing List Moderator
>
>
>
> On Jun 18, 2015, at 09:40, Steinsplitter Wiki <steinsplitter-wiki at live.com>
> wrote:
>
> @Martin Domdey: hi, ist exakt das selbe was in der vorherigen mail zu finden
> ist, ich erwarte mir von seiten der foundation eine genauere antwort. z.B.
> noch zwei stunden
>
> um ehrlich zu sein dürfte so etwas nicht passieren, toolslab ist stark
> genutzt.
>
> ________________________________
> From: animalia at gmx.net
> To: labs-l at lists.wikimedia.org
> Date: Thu, 18 Jun 2015 15:08:15 +0200
> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>
> Hey Steini, soll noch mehrere Stunden dauern laut den Technikern ...
>
> Gesendet: Donnerstag, 18. Juni 2015 um 14:55 Uhr
> Von: "Steinsplitter Wiki" <steinsplitter-wiki at live.com>
> An: "Wikimedia Labs" <labs-l at lists.wikimedia.org>
> Betreff: Re: [Labs-l] NFS outage in progress [UPDATE]
> Toolslabs will be back soon?  When approximately?
>
>> Date: Wed, 17 Jun 2015 23:09:11 -0400
>> From: marc at uberbox.org
>> To: abogott at wikimedia.org; labs-l at lists.wikimedia.org
>> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>>
>> On 15-06-17 09:29 PM, Andrew Bogott wrote:
>> > Coren is rebooting and fscking the system -- with luck it'll be up
>> > again within the hour.
>>
>> Fortune does not smile upon us; the fsck is taking a LONG time to
>> progress (over a 40T filesystem), but the repair goes apace.
>>
>> There is already indication of some lost data - though the failures are
>> likely to date from a past crash and may have gone unnoticed for some
>> time.
>>
>> Either way, we have a fairly recent (8-9 days) backup of the entire
>> filesystem so we may be able to recover lost or corrupted files if any
>> turn up.
>>
>> Worst case scenario is a catastrophic loss of the entire filesystem
>> (which does not seem likely at this time), in which case we will restore
>> the filesystem completely from the backup.
>>
>> The repair process will take at least a few more hours, during which I
>> will be keeping watch. I will update this list as soon as I have a
>> better assessment of the damage, if any.
>>
>> -- Marc
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
> _______________________________________________ Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
> _______________________________________________ Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
>
> _______________________________________________ Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
>
> _______________________________________________ Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>