[Labs-l] tools.wmflabs outage

[Petr Bena]

This is current load report for -dev:


Current Load
CRITICAL 2014-02-09 16:51:43 0d 1h 21m 34s 4/4 CRITICAL - load
average: 122.26, 118.04, 106.45

pretty much typical for processes hang waiting for IO (NFS outage).
The ssh might work, but because all /home folders are mounted on nfs
there is no way to use it unless you have access to account which
doesn't have home in /home

On Sun, Feb 9, 2014 at 5:50 PM, Petr Bena <benapetr at gmail.com> wrote:
> Pretty much nothing regular admins can fix, we indeed need someone
> from WMF with root keys / access to physical servers, because this
> seems to be issue on NFS again (IMHO) can't confirm that of course...
>
> Anyway, I can't even ssh there, I /could try/ to reboot some of the
> boxes like tools-dev but I am afraid it's not going to help anything.
>
> On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride at gmail.com> wrote:
>> All connections to tools.wmflabs is out, ssh, ftp, http, and https are all
>> non-functional. I am getting zero response on IRC, can someone with Coren's
>> mobile number give him a call?
>>
>> John
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>>