tools.wmflabs outage - Wikitech-l - lists.wikimedia.org

List overview All Threads
Download

tools.wmflabs outage

MeetBot now at your disposal in...

Bugzilla downtime: Wed Feb12 22:00...

John

9 Feb 2014 9 Feb '14

3:43 p.m.

All connections to tools.wmflabs is out, ssh, ftp, http, and https are all non-functional. I am getting zero response on IRC, can someone with Coren's mobile number give him a call? John

Reply

Show replies by date

Petr Bena

9 Feb 9 Feb

4:50 p.m.

New subject: [Labs-l] tools.wmflabs outage

Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course... Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything. On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride(a)gmail.com> wrote:

All connections to tools.wmflabs is out, ssh, ftp, http, and https are all non-functional. I am getting zero response on IRC, can someone with Coren's mobile number give him a call? John _______________________________________________ Labs-l mailing list Labs-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l

Reply

Petr Bena

4:54 p.m.

New subject: [Labs-l] tools.wmflabs outage

This is current load report for -dev: Current Load CRITICAL 2014-02-09 16:51:43 0d 1h 21m 34s 4/4 CRITICAL - load average: 122.26, 118.04, 106.45 pretty much typical for processes hang waiting for IO (NFS outage). The ssh might work, but because all /home folders are mounted on nfs there is no way to use it unless you have access to account which doesn't have home in /home On Sun, Feb 9, 2014 at 5:50 PM, Petr Bena <benapetr(a)gmail.com> wrote:

Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course... Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything. On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride(a)gmail.com> wrote: > All connections to tools.wmflabs is out, ssh, ftp, http, and https are all > non-functional. I am getting zero response on IRC, can someone with Coren's > mobile number give him a call? > > John > > _______________________________________________ > Labs-l mailing list > Labs-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/labs-l >

Reply

addshorewiki

4:55 p.m.

New subject: [Labs-l] tools.wmflabs outage

Coren is working on it :) XFS issues agin On 9 February 2014 17:50, Petr Bena <benapetr(a)gmail.com> wrote:

Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course... Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything. On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride(a)gmail.com> wrote:

All connections to tools.wmflabs is out, ssh, ftp, http, and https are

all

non-functional. I am getting zero response on IRC, can someone with

Coren's

mobile number give him a call? John _______________________________________________ Labs-l mailing list Labs-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Reply

John

4:55 p.m.

Does anyone have coren's number to let him know? He is the paid WMF contact for this. On Sunday, February 9, 2014, Petr Bena <benapetr(a)gmail.com> wrote:

Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course... Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything. On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride@gmail.com<javascript:;>> wrote:

All connections to tools.wmflabs is out, ssh, ftp, http, and https are

all

non-functional. I am getting zero response on IRC, can someone with

Coren's

mobile number give him a call? John _______________________________________________ Labs-l mailing list Labs-l(a)lists.wikimedia.org <javascript:;> https://lists.wikimedia.org/mailman/listinfo/labs-l

_______________________________________________ Labs-l mailing list Labs-l(a)lists.wikimedia.org <javascript:;> https://lists.wikimedia.org/mailman/listinfo/labs-l

Reply

addshorewiki

5:11 p.m.

New subject: [Labs-l] tools.wmflabs outage

Everything should be recovering now! Addshore On 9 February 2014 17:55, John <phoenixoverride(a)gmail.com> wrote:

Does anyone have coren's number to let him know? He is the paid WMF contact for this. On Sunday, February 9, 2014, Petr Bena <benapetr(a)gmail.com> wrote:

Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course... Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything. On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride(a)gmail.com> wrote:

All connections to tools.wmflabs is out, ssh, ftp, http, and https are

all

non-functional. I am getting zero response on IRC, can someone with

Coren's

mobile number give him a call? John _______________________________________________ Labs-l mailing list Labs-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l

_______________________________________________ Labs-l mailing list Labs-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l

_______________________________________________ Labs-l mailing list Labs-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l

Reply

3729

days inactive

3729

days old

wikitech-l@lists.wikimedia.org

Manage subscription

5 comments

3 participants

tags (0)

participants (3)

addshorewiki
John
Petr Bena