September 2019 - Cloud-admin-feed

** PROBLEM alert - labstore1006/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 29 Sep '19

29 Sep '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1006 Address: 208.80.154.7 State: CRITICAL Date/Time: Sun Sept 29 12:26:41 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=wmcs instance=labstore1006:9100 job=node site=eqiad

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1024/MegaRAID is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Sep '19

27 Sep '19

Notification Type: ACKNOWLEDGEMENT Service: MegaRAID Host: cloudvirt1024 Address: 10.64.20.43 State: CRITICAL Date/Time: Fri Sept 27 13:45:16 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Acknowledged by nagiosadmin: RAID handler auto-ack: https://phabricator.wikimedia.org/T234044 Additional Info: CRITICAL: 1 failed LD(s) (Offline)

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1024/MegaRAID is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Sep '19

27 Sep '19

Notification Type: ACKNOWLEDGEMENT Service: MegaRAID Host: cloudvirt1024 Address: 10.64.20.43 State: CRITICAL Date/Time: Fri Sept 27 08:41:27 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Acknowledged by nagiosadmin: RAID handler auto-ack: https://phabricator.wikimedia.org/T234018 Additional Info: CRITICAL: 1 failed LD(s) (Degraded)

1 0

** RECOVERY alert - labstore1006/High 1m load average is OK **
by nagios＠icinga1001.wikimedia.org 27 Sep '19

27 Sep '19

Notification Type: RECOVERY Service: High 1m load average Host: labstore1006 Address: 208.80.154.7 State: OK Date/Time: Fri Sept 27 07:58:49 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: All metrics within thresholds.

1 0

** PROBLEM alert - labstore1006/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Sep '19

27 Sep '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1006 Address: 208.80.154.7 State: CRITICAL Date/Time: Fri Sept 27 07:26:05 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=wmcs instance=labstore1006:9100 job=node site=eqiad

1 0

** PROBLEM alert - labstore1006/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Sep '19

27 Sep '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1006 Address: 208.80.154.7 State: CRITICAL Date/Time: Fri Sept 27 05:18:13 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=wmcs instance=labstore1006:9100 job=node site=eqiad

1 0

** RECOVERY alert - labsdb1010/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 26 Sep '19

26 Sep '19

Notification Type: RECOVERY Service: Check systemd state Host: labsdb1010 Address: 10.64.37.23 State: OK Date/Time: Thu Sept 26 20:33:18 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - labsdb1010/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 26 Sep '19

26 Sep '19

Notification Type: PROBLEM Service: Check systemd state Host: labsdb1010 Address: 10.64.37.23 State: CRITICAL Date/Time: Thu Sept 26 20:10:48 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** RECOVERY alert - labsdb1011/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 26 Sep '19

26 Sep '19

Notification Type: RECOVERY Service: Check systemd state Host: labsdb1011 Address: 10.64.37.24 State: OK Date/Time: Thu Sept 26 20:02:20 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - labsdb1011/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 26 Sep '19

26 Sep '19

Notification Type: PROBLEM Service: Check systemd state Host: labsdb1011 Address: 10.64.37.24 State: CRITICAL Date/Time: Thu Sept 26 19:54:14 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed September 2019