October 2019 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - labstore1005/Ensure NFS exports are maintained for new instances with NFS is CRITICAL **
by nagios＠icinga1001.wikimedia.org 07 Oct '19

07 Oct '19

Notification Type: PROBLEM Service: Ensure NFS exports are maintained for new instances with NFS Host: labstore1005 Address: 10.64.37.20 State: CRITICAL Date/Time: Mon Oct 7 14:26:44 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Acknowledged by : Additional Info: CRITICAL - Expecting active but unit nfs-exportd is activating

1 0

** PROBLEM alert - labstore1004/Ensure NFS exports are maintained for new instances with NFS is CRITICAL **
by nagios＠icinga1001.wikimedia.org 07 Oct '19

07 Oct '19

Notification Type: PROBLEM Service: Ensure NFS exports are maintained for new instances with NFS Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Mon Oct 7 14:25:54 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Acknowledged by : Additional Info: CRITICAL - Expecting active but unit nfs-exportd is activating

1 0

** PROBLEM alert - labstore1004/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 07 Oct '19

07 Oct '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Mon Oct 7 13:19:19 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=labsnfs instance=labstore1004:9100 job=node site=eqiad

1 0

** RECOVERY alert - labstore1004/High 1m load average is OK **
by nagios＠icinga1001.wikimedia.org 07 Oct '19

07 Oct '19

Notification Type: RECOVERY Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Mon Oct 7 05:29:09 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: All metrics within thresholds.

1 0

** PROBLEM alert - labstore1004/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 07 Oct '19

07 Oct '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Mon Oct 7 04:36:57 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=labsnfs instance=labstore1004:9100 job=node site=eqiad

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is OK **
by nagios＠icinga1001.wikimedia.org 07 Oct '19

07 Oct '19

Notification Type: RECOVERY Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Mon Oct 7 02:03:51 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.121 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 07 Oct '19

07 Oct '19

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Oct 7 01:58:29 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.161 second response time

1 0

** RECOVERY alert - labstore1004/High 1m load average is OK **
by nagios＠icinga1001.wikimedia.org 06 Oct '19

06 Oct '19

Notification Type: RECOVERY Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Sun Oct 6 22:42:13 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: All metrics within thresholds.

1 0

** PROBLEM alert - labstore1004/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 06 Oct '19

06 Oct '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Sun Oct 6 22:27:37 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=labsnfs instance=labstore1004:9100 job=node site=eqiad

1 0

** RECOVERY alert - labsdb1010/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 06 Oct '19

06 Oct '19

Notification Type: RECOVERY Service: Check systemd state Host: labsdb1010 Address: 10.64.37.23 State: OK Date/Time: Sun Oct 6 19:03:05 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed October 2019