November 2019 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - cloudmetrics1002/graphite-labs.wikimedia.org render is CRITICAL **
by nagios＠icinga1001.wikimedia.org 28 Nov '19

28 Nov '19

Notification Type: PROBLEM Service: graphite-labs.wikimedia.org render Host: cloudmetrics1002 Address: 10.64.4.15 State: CRITICAL Date/Time: Thu Nov 28 18:20:00 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Graphite%23Operations_troubleshooting Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 129 bytes in 0.001 second response time

1 0

** PROBLEM alert - cloudmetrics1002/graphite-labs.wikimedia.org api is CRITICAL **
by nagios＠icinga1001.wikimedia.org 28 Nov '19

28 Nov '19

Notification Type: PROBLEM Service: graphite-labs.wikimedia.org api Host: cloudmetrics1002 Address: 10.64.4.15 State: CRITICAL Date/Time: Thu Nov 28 18:18:54 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Graphite%23Operations_troubleshooting Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 104 bytes in 0.001 second response time

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is OK **
by nagios＠icinga1001.wikimedia.org 27 Nov '19

27 Nov '19

Notification Type: RECOVERY Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Wed Nov 27 20:57:55 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.119 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Nov '19

27 Nov '19

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Wed Nov 27 20:52:29 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.105 second response time

1 0

** RECOVERY alert - labstore1007/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 27 Nov '19

27 Nov '19

Notification Type: RECOVERY Service: Check systemd state Host: labstore1007 Address: 208.80.155.106 State: OK Date/Time: Wed Nov 27 17:19:26 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - labstore1007/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Nov '19

27 Nov '19

Notification Type: PROBLEM Service: Check systemd state Host: labstore1007 Address: 208.80.155.106 State: CRITICAL Date/Time: Wed Nov 27 17:08:11 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** RECOVERY alert - cloudbackup2002/Check the last execution of block_sync-misc-project is OK **
by nagios＠icinga1001.wikimedia.org 26 Nov '19

26 Nov '19

Notification Type: RECOVERY Service: Check the last execution of block_sync-misc-project Host: cloudbackup2002 Address: 10.192.32.186 State: OK Date/Time: Tue Nov 26 20:19:42 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time… Acknowledged by : Additional Info: OK: Status of the systemd unit block_sync-misc-project

1 0

** RECOVERY alert - cloudbackup2002/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 26 Nov '19

26 Nov '19

Notification Type: RECOVERY Service: Check systemd state Host: cloudbackup2002 Address: 10.192.32.186 State: OK Date/Time: Tue Nov 26 20:14:30 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** ACKNOWLEDGEMENT alert - cloudbackup2002/Check the last execution of block_sync-misc-project is CRITICAL **
by nagios＠icinga1001.wikimedia.org 26 Nov '19

26 Nov '19

Notification Type: ACKNOWLEDGEMENT Service: Check the last execution of block_sync-misc-project Host: cloudbackup2002 Address: 10.192.32.186 State: CRITICAL Date/Time: Tue Nov 26 20:14:12 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time… Acknowledged by Bstorm: The target volume is still mounted for testing. Nothing to worry about. Additional Info: CRITICAL: Status of the systemd unit block_sync-misc-project

1 0

** ACKNOWLEDGEMENT alert - cloudbackup2002/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 26 Nov '19

26 Nov '19

Notification Type: ACKNOWLEDGEMENT Service: Check systemd state Host: cloudbackup2002 Address: 10.192.32.186 State: CRITICAL Date/Time: Tue Nov 26 20:14:12 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by Bstorm: The target volume is still mounted for testing. Nothing to worry about. Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed November 2019