January 2020 - Cloud-admin-feed - lists.wikimedia.org

** RECOVERY alert - cloudcontrol1003/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 12 Jan '20

12 Jan '20

Notification Type: RECOVERY Service: Check systemd state Host: cloudcontrol1003 Address: 208.80.154.23 State: OK Date/Time: Mon Jan 13 02:59:35 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - cloudcontrol1004/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Jan '20

12 Jan '20

Notification Type: PROBLEM Service: Check systemd state Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Mon Jan 13 02:58:16 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** PROBLEM alert - cloudcontrol1003/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Jan '20

12 Jan '20

Notification Type: PROBLEM Service: Check systemd state Host: cloudcontrol1003 Address: 208.80.154.23 State: CRITICAL Date/Time: Mon Jan 13 02:57:46 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1013/HP RAID is CRITICAL **
by nagios＠icinga1001.wikimedia.org 10 Jan '20

10 Jan '20

Notification Type: ACKNOWLEDGEMENT Service: HP RAID Host: cloudvirt1013 Address: 10.64.20.29 State: CRITICAL Date/Time: Fri Jan 10 22:18:27 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_… Acknowledged by nagiosadmin: RAID handler auto-ack: https://phabricator.wikimedia.org/T242472 Additional Info: CRITICAL: Slot 0: OK: 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:1:5, 2I:1:6 - Controller: OK - Battery count: 0

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s etcd nodes are healthy is OK **
by nagios＠icinga1001.wikimedia.org 08 Jan '20

08 Jan '20

Notification Type: RECOVERY Service: toolschecker: All k8s etcd nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Wed Jan 8 19:01:29 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 1.454 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s etcd nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 08 Jan '20

08 Jan '20

Notification Type: PROBLEM Service: toolschecker: All k8s etcd nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Wed Jan 8 18:56:09 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/k8s - 177 bytes in 1.754 second response time

1 0

** RECOVERY alert - cloudcontrol1003/tools project instance distribution is OK **
by nagios＠icinga1001.wikimedia.org 06 Jan '20

06 Jan '20

Notification Type: RECOVERY Service: tools project instance distribution Host: cloudcontrol1003 Address: 208.80.154.23 State: OK Date/Time: Tue Jan 7 00:08:19 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: OK: All critical instances are spread out enough

1 0

** PROBLEM alert - cloudcontrol1003/tools project instance distribution is CRITICAL **
by nagios＠icinga1001.wikimedia.org 06 Jan '20

06 Jan '20

Notification Type: PROBLEM Service: tools project instance distribution Host: cloudcontrol1003 Address: 208.80.154.23 State: CRITICAL Date/Time: Mon Jan 6 23:41:23 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: CRITICAL: sgewebgrid-lighttpd class instances not spread out enough

1 0

** RECOVERY alert - cloudcontrol1003/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 06 Jan '20

06 Jan '20

Notification Type: RECOVERY Service: Check systemd state Host: cloudcontrol1003 Address: 208.80.154.23 State: OK Date/Time: Mon Jan 6 13:35:52 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - cloudcontrol1003/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 06 Jan '20

06 Jan '20

Notification Type: PROBLEM Service: Check systemd state Host: cloudcontrol1003 Address: 208.80.154.23 State: CRITICAL Date/Time: Mon Jan 6 13:34:04 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed January 2020