April 2020 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - labstore1007/High 1m load average is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: High 1m load average Host: labstore1007 Address: 208.80.155.106 State: CRITICAL Date/Time: Wed Apr 22 13:56:19 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=wmcs instance=labstore1007:9100 job=node site=eqiad

4 years, 1 month

1
0
0 0

** PROBLEM alert - labstore1007/High 1m load average is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: High 1m load average Host: labstore1007 Address: 208.80.155.106 State: CRITICAL Date/Time: Wed Apr 22 13:11:55 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=wmcs instance=labstore1007:9100 job=node site=eqiad

4 years, 1 month

1
0
0 0

** PROBLEM alert - labstore1007/High 1m load average is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: High 1m load average Host: labstore1007 Address: 208.80.155.106 State: CRITICAL Date/Time: Wed Apr 22 12:58:57 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=wmcs instance=labstore1007:9100 job=node site=eqiad

4 years, 1 month

1
0
0 0

** RECOVERY alert - cloudcontrol1004/tools project instance distribution is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: tools project instance distribution Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Tue Apr 21 22:51:44 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: OK: All critical instances are spread out enough

4 years, 1 month

1
0
0 0

** RECOVERY alert - cloudcontrol1003/tools project instance distribution is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: tools project instance distribution Host: cloudcontrol1003 Address: 208.80.154.23 State: OK Date/Time: Tue Apr 21 22:50:02 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: OK: All critical instances are spread out enough

4 years, 1 month

1
0
0 0

** PROBLEM alert - cloudcontrol1003/tools project instance distribution is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: tools project instance distribution Host: cloudcontrol1003 Address: 208.80.154.23 State: CRITICAL Date/Time: Tue Apr 21 22:44:28 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: CRITICAL: sgewebgrid-lighttpd class instances not spread out enough

4 years, 1 month

1
0
0 0

** PROBLEM alert - cloudcontrol1004/tools project instance distribution is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: tools project instance distribution Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Tue Apr 21 22:44:18 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: CRITICAL: sgewebgrid-lighttpd class instances not spread out enough

4 years, 1 month

1
0
0 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Tue Apr 21 22:07:54 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.052 second response time

4 years, 1 month

1
0
0 0

** RECOVERY alert - cloudvirt1004/Check systemd state is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: Check systemd state Host: cloudvirt1004 Address: 10.64.20.22 State: OK Date/Time: Tue Apr 21 22:05:20 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

4 years, 1 month

1
0
0 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Tue Apr 21 21:50:24 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.087 second response time

4 years, 1 month

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed April 2020