December 2019 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - labstore1006/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Dec '19

17 Dec '19

Notification Type: PROBLEM Service: Check systemd state Host: labstore1006 Address: 208.80.154.7 State: CRITICAL Date/Time: Tue Dec 17 15:57:33 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Dec '19

17 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Tue Dec 17 02:50:20 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Bstorm: Theres no issue on the old k8s cluster, but tools checker needs an update T240923 Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 233 bytes in 0.003 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Dec '19

17 Dec '19

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Tue Dec 17 02:35:47 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 233 bytes in 0.004 second response time

1 0

** RECOVERY alert - cloudcontrol1004/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 15 Dec '19

15 Dec '19

Notification Type: RECOVERY Service: Check systemd state Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Sun Dec 15 02:21:15 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - cloudcontrol1004/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 15 Dec '19

15 Dec '19

Notification Type: PROBLEM Service: Check systemd state Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Sun Dec 15 02:19:27 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** RECOVERY alert - cloudvirt1023/ensure kvm processes are running is OK **
by nagios＠icinga1001.wikimedia.org 15 Dec '19

15 Dec '19

Notification Type: RECOVERY Service: ensure kvm processes are running Host: cloudvirt1023 Address: 10.64.20.42 State: OK Date/Time: Sun Dec 15 02:06:15 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS OK: 1 process with regex args qemu-system-x86_64

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1023/ensure kvm processes are running is CRITICAL **
by nagios＠icinga1001.wikimedia.org 15 Dec '19

15 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: ensure kvm processes are running Host: cloudvirt1023 Address: 10.64.20.42 State: CRITICAL Date/Time: Sun Dec 15 02:03:56 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by andrew bogott: Ill investigate Additional Info: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64

1 0

** PROBLEM alert - cloudvirt1023/ensure kvm processes are running is CRITICAL **
by nagios＠icinga1001.wikimedia.org 15 Dec '19

15 Dec '19

Notification Type: PROBLEM Service: ensure kvm processes are running Host: cloudvirt1023 Address: 10.64.20.42 State: CRITICAL Date/Time: Sun Dec 15 02:02:37 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64

1 0

** RECOVERY alert - labstore1004/High 1m load average is OK **
by nagios＠icinga1001.wikimedia.org 14 Dec '19

14 Dec '19

Notification Type: RECOVERY Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Sat Dec 14 02:52:53 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: All metrics within thresholds.

1 0

** PROBLEM alert - labstore1004/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 14 Dec '19

14 Dec '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Sat Dec 14 02:33:07 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=labsnfs instance=labstore1004:9100 job=node site=eqiad

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed December 2019