February 2020 - Cloud-admin-feed - lists.wikimedia.org

Host DOWN alert for labstore1004!
by nagios＠icinga1001.wikimedia.org 27 Feb '20

27 Feb '20

Notification Type: PROBLEM Host: labstore1004 State: DOWN Address: 10.64.37.19 Info: PING CRITICAL - Packet loss = 100% Date/Time: Thu Feb 27 17:05:16 UTC 2020 Acknowledged by :

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: NFS read/writeable on labs instances is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Feb '20

27 Feb '20

Notification Type: PROBLEM Service: toolschecker: NFS read/writeable on labs instances Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Feb 27 17:02:52 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/nfs/home - 340 bytes in 60.012 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: check mtime mod from tools cron job is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Feb '20

27 Feb '20

Notification Type: PROBLEM Service: toolschecker: check mtime mod from tools cron job Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Feb 27 17:02:30 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/cron - 340 bytes in 60.010 second response time

1 0

** PROBLEM alert - labstore1005/NFS port is open on cluster IP is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Feb '20

27 Feb '20

Notification Type: PROBLEM Service: NFS port is open on cluster IP Host: labstore1005 Address: 10.64.37.20 State: CRITICAL Date/Time: Thu Feb 27 17:02:12 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore Acknowledged by : Additional Info: CRITICAL - Socket timeout after 2 seconds

1 0

** RECOVERY alert - labstore1004/High 1m load average is OK **
by nagios＠icinga1001.wikimedia.org 27 Feb '20

27 Feb '20

Notification Type: RECOVERY Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Thu Feb 27 03:26:46 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: All metrics within thresholds.

1 0

** PROBLEM alert - labstore1004/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Feb '20

27 Feb '20

Notification Type: PROBLEM Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Thu Feb 27 02:37:01 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=labsnfs instance=labstore1004:9100 job=node site=eqiad

1 0

** PROBLEM alert - labstore1004/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Feb '20

27 Feb '20

Notification Type: PROBLEM Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Thu Feb 27 00:02:02 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=labsnfs instance=labstore1004:9100 job=node site=eqiad

1 0

** PROBLEM alert - labstore1004/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Feb '20

27 Feb '20

Notification Type: PROBLEM Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Wed Feb 26 23:44:24 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=labsnfs instance=labstore1004:9100 job=node site=eqiad

1 0

** RECOVERY alert - cloudvirt1014/HP RAID is OK **
by nagios＠icinga1001.wikimedia.org 26 Feb '20

26 Feb '20

Notification Type: RECOVERY Service: HP RAID Host: cloudvirt1014 Address: 10.64.20.30 State: OK Date/Time: Wed Feb 26 22:19:24 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_… Acknowledged by : Additional Info: OK: Slot 0: OK: 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:1:5, 2I:1:6 - Controller: OK - Battery/Capacitor: OK

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: check mtime mod from tools cron job is OK **
by nagios＠icinga1001.wikimedia.org 26 Feb '20

26 Feb '20

Notification Type: RECOVERY Service: toolschecker: check mtime mod from tools cron job Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Wed Feb 26 20:57:09 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.007 second response time

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed February 2020