[Labs-l] tools.wmflabs outage

[Petr Bena]

This was XFS (filesystem) issue on storage used by NFS, so it wasn't
directly a failure caused by NFS server

On Sun, Feb 9, 2014 at 6:37 PM, Maximilian Doerr <cybernet678 at yahoo.com> wrote:
> Wasn't NFS setup so this wouldn't happen again? :p
>
> Gesendet von Maximilian's iPad.
> (Sent from Maximilian's iPad.)
>
> On Feb 9, 2014, at 12:11, addshorewiki <addshorewiki at gmail.com> wrote:
>
> Everything should be recovering now!
>
> Addshore
>
>
> On 9 February 2014 17:55, John <phoenixoverride at gmail.com> wrote:
>>
>> Does anyone have coren's number to let him know? He is the paid WMF
>> contact for this.
>>
>>
>> On Sunday, February 9, 2014, Petr Bena <benapetr at gmail.com> wrote:
>>>
>>> Pretty much nothing regular admins can fix, we indeed need someone
>>> from WMF with root keys / access to physical servers, because this
>>> seems to be issue on NFS again (IMHO) can't confirm that of course...
>>>
>>> Anyway, I can't even ssh there, I /could try/ to reboot some of the
>>> boxes like tools-dev but I am afraid it's not going to help anything.
>>>
>>> On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride at gmail.com> wrote:
>>> > All connections to tools.wmflabs is out, ssh, ftp, http, and https are
>>> > all
>>> > non-functional. I am getting zero response on IRC, can someone with
>>> > Coren's
>>> > mobile number give him a call?
>>> >
>>> > John
>>> >
>>> > _______________________________________________
>>> > Labs-l mailing list
>>> > Labs-l at lists.wikimedia.org
>>> > https://lists.wikimedia.org/mailman/listinfo/labs-l
>>> >
>>>
>>> _______________________________________________
>>> Labs-l mailing list
>>> Labs-l at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>