[Labs-l] NFS outage in progress [UPDATE]

Thu Jun 18 14:09:14 UTC 2015

These are of course just my thoughts. No references to back this :)

On Thu, Jun 18, 2015 at 4:07 PM, Petr Bena <benapetr at gmail.com> wrote:
> That is actually not that easy to decide on. The biggest community are
> wikipedia readers. They don't really care about labs at all. Second
> biggest community are editors, they don't care so much about labs
> either.
>
> Developer community is not that big and most of money comes from
> readers. If there was a way for donors to choose which part of
> wikimedia project they would like to support with their money, I doubt
> there would be many of them who would actually pick "labs".
>
> On Thu, Jun 18, 2015 at 4:03 PM, Bene* <benestar.wikimedia at gmail.com> wrote:
>> I think that's exactly the point where WMF is to blame. Labs is a powerful
>> place on which lots of communities rely. We need our bots and tools do work
>> efficiently, otherwise it is hard to do cool things on wikis.
>>
>> So the point is, if "it doesn't have so big budget" this is a big mistake
>> and the foundation should consider to change that. You see, community gets
>> angry if their tools don't work.
>>
>> Best regards
>> Bene
>>
>>
>> Am 18.06.2015 um 16:01 schrieb Petr Bena:
>>>
>>> Kdyz tady bude kazdy mluvit svym jazykem tak si moc nepokecame...
>>>
>>> To be fair, labs do work - most of the time, and afaik it doesn't have
>>> so big budget, so blaming the foundation is not very nice. It was
>>> always meant for development. Ops are actually working hard on it to
>>> be as much stable as possible.
>>>
>>>
>>> On Thu, Jun 18, 2015 at 3:49 PM, Maximilian Doerr
>>> <maximilian.doerr at gmail.com> wrote:
>>>>
>>>> Any ETA when I can SSH back in and restore my bots?
>>>>
>>>> Cyberpower678
>>>> English Wikipedia Account Creation Team
>>>> Mailing List Moderator
>>>>
>>>>
>>>>
>>>> On Jun 18, 2015, at 09:48, Steinsplitter Wiki
>>>> <steinsplitter-wiki at live.com>
>>>> wrote:
>>>>
>>>> the foundation isn't able to provide a service that works in real life.
>>>>
>>>> ________________________________
>>>> From: maximilian.doerr at gmail.com
>>>> Date: Thu, 18 Jun 2015 09:47:07 -0400
>>>> To: labs-l at lists.wikimedia.org
>>>> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>>>>
>>>> Mist.  Ich bekomme wieder beschwerte dass meine Bots schon wieder nicht
>>>> funktionieren. :/
>>>>
>>>> Cyberpower678
>>>> English Wikipedia Account Creation Team
>>>> Mailing List Moderator
>>>>
>>>>
>>>>
>>>> On Jun 18, 2015, at 09:45, Steinsplitter Wiki
>>>> <steinsplitter-wiki at live.com>
>>>> wrote:
>>>>
>>>>>> Hat was hier los ist damit was zum tun?
>>>>
>>>> ja :(
>>>>
>>>> ________________________________
>>>> From: maximilian.doerr at gmail.com
>>>> Date: Thu, 18 Jun 2015 09:41:53 -0400
>>>> To: labs-l at lists.wikimedia.org
>>>> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>>>>
>>>> Ich kann weder einloggen oder meine Bot Seiten aufrufen.  Hat was hier
>>>> los
>>>> ist damit was zum tun?  ACC funktioniert auch nicht.
>>>>
>>>> Cyberpower678
>>>> English Wikipedia Account Creation Team
>>>> Mailing List Moderator
>>>>
>>>>
>>>>
>>>> On Jun 18, 2015, at 09:40, Steinsplitter Wiki
>>>> <steinsplitter-wiki at live.com>
>>>> wrote:
>>>>
>>>> @Martin Domdey: hi, ist exakt das selbe was in der vorherigen mail zu
>>>> finden
>>>> ist, ich erwarte mir von seiten der foundation eine genauere antwort.
>>>> z.B.
>>>> noch zwei stunden
>>>>
>>>> um ehrlich zu sein dürfte so etwas nicht passieren, toolslab ist stark
>>>> genutzt.
>>>>
>>>> ________________________________
>>>> From: animalia at gmx.net
>>>> To: labs-l at lists.wikimedia.org
>>>> Date: Thu, 18 Jun 2015 15:08:15 +0200
>>>> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>>>>
>>>> Hey Steini, soll noch mehrere Stunden dauern laut den Technikern ...
>>>>
>>>> Gesendet: Donnerstag, 18. Juni 2015 um 14:55 Uhr
>>>> Von: "Steinsplitter Wiki" <steinsplitter-wiki at live.com>
>>>> An: "Wikimedia Labs" <labs-l at lists.wikimedia.org>
>>>> Betreff: Re: [Labs-l] NFS outage in progress [UPDATE]
>>>> Toolslabs will be back soon?  When approximately?
>>>>
>>>>> Date: Wed, 17 Jun 2015 23:09:11 -0400
>>>>> From: marc at uberbox.org
>>>>> To: abogott at wikimedia.org; labs-l at lists.wikimedia.org
>>>>> Subject: Re: [Labs-l] NFS outage in progress [UPDATE]
>>>>>
>>>>> On 15-06-17 09:29 PM, Andrew Bogott wrote:
>>>>>>
>>>>>> Coren is rebooting and fscking the system -- with luck it'll be up
>>>>>> again within the hour.
>>>>>
>>>>> Fortune does not smile upon us; the fsck is taking a LONG time to
>>>>> progress (over a 40T filesystem), but the repair goes apace.
>>>>>
>>>>> There is already indication of some lost data - though the failures are
>>>>> likely to date from a past crash and may have gone unnoticed for some
>>>>> time.
>>>>>
>>>>> Either way, we have a fairly recent (8-9 days) backup of the entire
>>>>> filesystem so we may be able to recover lost or corrupted files if any
>>>>> turn up.
>>>>>
>>>>> Worst case scenario is a catastrophic loss of the entire filesystem
>>>>> (which does not seem likely at this time), in which case we will restore
>>>>> the filesystem completely from the backup.
>>>>>
>>>>> The repair process will take at least a few more hours, during which I
>>>>> will be keeping watch. I will update this list as soon as I have a
>>>>> better assessment of the damage, if any.
>>>>>
>>>>> -- Marc
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Labs-l mailing list
>>>>> Labs-l at lists.wikimedia.org
>>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>
>>>> _______________________________________________ Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>
>>>> _______________________________________________ Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>> _______________________________________________
>>>> Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>
>>>>
>>>>
>>>> _______________________________________________ Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>> _______________________________________________
>>>> Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>
>>>>
>>>>
>>>> _______________________________________________ Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>> _______________________________________________
>>>> Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l