[Labs-l] tools.wmflabs outage

[Daniel Mietchen]

Labs seems off again.


On Sun, Feb 9, 2014 at 6:48 PM, Petr Bena <benapetr at gmail.com> wrote:
> This was XFS (filesystem) issue on storage used by NFS, so it wasn't
> directly a failure caused by NFS server
>
> On Sun, Feb 9, 2014 at 6:37 PM, Maximilian Doerr <cybernet678 at yahoo.com> wrote:
>> Wasn't NFS setup so this wouldn't happen again? :p
>>
>> Gesendet von Maximilian's iPad.
>> (Sent from Maximilian's iPad.)
>>
>> On Feb 9, 2014, at 12:11, addshorewiki <addshorewiki at gmail.com> wrote:
>>
>> Everything should be recovering now!
>>
>> Addshore
>>
>>
>> On 9 February 2014 17:55, John <phoenixoverride at gmail.com> wrote:
>>>
>>> Does anyone have coren's number to let him know? He is the paid WMF
>>> contact for this.
>>>
>>>
>>> On Sunday, February 9, 2014, Petr Bena <benapetr at gmail.com> wrote:
>>>>
>>>> Pretty much nothing regular admins can fix, we indeed need someone
>>>> from WMF with root keys / access to physical servers, because this
>>>> seems to be issue on NFS again (IMHO) can't confirm that of course...
>>>>
>>>> Anyway, I can't even ssh there, I /could try/ to reboot some of the
>>>> boxes like tools-dev but I am afraid it's not going to help anything.
>>>>
>>>> On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride at gmail.com> wrote:
>>>> > All connections to tools.wmflabs is out, ssh, ftp, http, and https are
>>>> > all
>>>> > non-functional. I am getting zero response on IRC, can someone with
>>>> > Coren's
>>>> > mobile number give him a call?
>>>> >
>>>> > John
>>>> >
>>>> > _______________________________________________
>>>> > Labs-l mailing list
>>>> > Labs-l at lists.wikimedia.org
>>>> > https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>> >
>>>>
>>>> _______________________________________________
>>>> Labs-l mailing list
>>>> Labs-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l