All connections to tools.wmflabs is out, ssh, ftp, http, and https are all non-functional. I am getting zero response on IRC, can someone with Coren's mobile number give him a call?
John
Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course...
Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything.
On Sun, Feb 9, 2014 at 4:43 PM, John phoenixoverride@gmail.com wrote:
All connections to tools.wmflabs is out, ssh, ftp, http, and https are all non-functional. I am getting zero response on IRC, can someone with Coren's mobile number give him a call?
John
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
This is current load report for -dev:
Current Load CRITICAL 2014-02-09 16:51:43 0d 1h 21m 34s 4/4 CRITICAL - load average: 122.26, 118.04, 106.45
pretty much typical for processes hang waiting for IO (NFS outage). The ssh might work, but because all /home folders are mounted on nfs there is no way to use it unless you have access to account which doesn't have home in /home
On Sun, Feb 9, 2014 at 5:50 PM, Petr Bena benapetr@gmail.com wrote:
Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course...
Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything.
On Sun, Feb 9, 2014 at 4:43 PM, John phoenixoverride@gmail.com wrote:
All connections to tools.wmflabs is out, ssh, ftp, http, and https are all non-functional. I am getting zero response on IRC, can someone with Coren's mobile number give him a call?
John
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Coren is working on it :)
XFS issues agin
On 9 February 2014 17:50, Petr Bena benapetr@gmail.com wrote:
Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course...
Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything.
On Sun, Feb 9, 2014 at 4:43 PM, John phoenixoverride@gmail.com wrote:
All connections to tools.wmflabs is out, ssh, ftp, http, and https are
all
non-functional. I am getting zero response on IRC, can someone with
Coren's
mobile number give him a call?
John
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Does anyone have coren's number to let him know? He is the paid WMF contact for this.
On Sunday, February 9, 2014, Petr Bena benapetr@gmail.com wrote:
Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course...
Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything.
On Sun, Feb 9, 2014 at 4:43 PM, John <phoenixoverride@gmail.comjavascript:;> wrote:
All connections to tools.wmflabs is out, ssh, ftp, http, and https are
all
non-functional. I am getting zero response on IRC, can someone with
Coren's
mobile number give him a call?
John
Labs-l mailing list Labs-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/labs-l
Everything should be recovering now!
Addshore
On 9 February 2014 17:55, John phoenixoverride@gmail.com wrote:
Does anyone have coren's number to let him know? He is the paid WMF contact for this.
On Sunday, February 9, 2014, Petr Bena benapetr@gmail.com wrote:
Pretty much nothing regular admins can fix, we indeed need someone from WMF with root keys / access to physical servers, because this seems to be issue on NFS again (IMHO) can't confirm that of course...
Anyway, I can't even ssh there, I /could try/ to reboot some of the boxes like tools-dev but I am afraid it's not going to help anything.
On Sun, Feb 9, 2014 at 4:43 PM, John phoenixoverride@gmail.com wrote:
All connections to tools.wmflabs is out, ssh, ftp, http, and https are
all
non-functional. I am getting zero response on IRC, can someone with
Coren's
mobile number give him a call?
John
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
Labs-l mailing list Labs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/labs-l
wikitech-l@lists.wikimedia.org