August 2019 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 15 Aug '19

15 Aug '19

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Aug 15 10:08:09 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.103 second response time

1 0

** PROBLEM alert - labstore1007/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 15 Aug '19

15 Aug '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1007 Address: 208.80.155.106 State: CRITICAL Date/Time: Thu Aug 15 09:47:35 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: cluster=wmcs instance=labstore1007:9100 job=node site=eqiad

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1024/Disk space is CRITICAL **
by nagios＠icinga1001.wikimedia.org 14 Aug '19

14 Aug '19

Notification Type: ACKNOWLEDGEMENT Service: Disk space Host: cloudvirt1024 Address: 10.64.20.43 State: CRITICAL Date/Time: Wed Aug 14 00:58:04 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=cloudvi… Additional Info: DISK CRITICAL - /var/lib/nova/instances is not accessible: Input/output error

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1024/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 14 Aug '19

14 Aug '19

Notification Type: ACKNOWLEDGEMENT Service: Check systemd state Host: cloudvirt1024 Address: 10.64.20.43 State: CRITICAL Date/Time: Wed Aug 14 00:54:29 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** PROBLEM alert - cloudvirt1024/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 14 Aug '19

14 Aug '19

Notification Type: PROBLEM Service: Check systemd state Host: cloudvirt1024 Address: 10.64.20.43 State: CRITICAL Date/Time: Wed Aug 14 00:51:19 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** PROBLEM alert - cloudvirt1024/Disk space is CRITICAL **
by nagios＠icinga1001.wikimedia.org 14 Aug '19

14 Aug '19

Notification Type: PROBLEM Service: Disk space Host: cloudvirt1024 Address: 10.64.20.43 State: CRITICAL Date/Time: Wed Aug 14 00:49:33 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=cloudvi… Additional Info: DISK CRITICAL - /var/lib/nova/instances is not accessible: Input/output error

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1024/MegaRAID is CRITICAL **
by nagios＠icinga1001.wikimedia.org 13 Aug '19

13 Aug '19

Notification Type: ACKNOWLEDGEMENT Service: MegaRAID Host: cloudvirt1024 Address: 10.64.20.43 State: CRITICAL Date/Time: Tue Aug 13 18:30:59 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Additional Info: CRITICAL: 1 failed LD(s) (Degraded)

1 0

** PROBLEM alert - cloudvirt1024/MegaRAID is CRITICAL **
by nagios＠icinga1001.wikimedia.org 13 Aug '19

13 Aug '19

Notification Type: PROBLEM Service: MegaRAID Host: cloudvirt1024 Address: 10.64.20.43 State: CRITICAL Date/Time: Tue Aug 13 18:30:58 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Additional Info: CRITICAL: 1 failed LD(s) (Degraded)

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: showmount succeeds on a labs instance is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Aug '19

12 Aug '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: showmount succeeds on a labs instance Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Aug 12 20:47:05 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/nfs/secondary_cluster_showmount - 177 bytes in 0.040 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: showmount succeeds on a labs instance is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Aug '19

12 Aug '19

Notification Type: PROBLEM Service: toolschecker: showmount succeeds on a labs instance Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Aug 12 20:44:22 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/nfs/secondary_cluster_showmount - 177 bytes in 0.024 second response time

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed August 2019