- Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 11:58:00 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.116 second response time

1 0

** RECOVERY alert - labstore1004/High 1m load average is OK **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: RECOVERY Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Thu Dec 12 11:52:58 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: All metrics within thresholds.

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is OK **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: RECOVERY Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Thu Dec 12 11:40:40 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.123 second response time

1 0

** PROBLEM alert - labstore1004/High 1m load average is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: PROBLEM Service: High 1m load average Host: labstore1004 Address: 10.64.37.19 State: CRITICAL Date/Time: Thu Dec 12 11:40:20 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: cluster=labsnfs instance=labstore1004:9100 job=node site=eqiad

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 11:34:52 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.190 second response time

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: Redis set/get is OK **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: RECOVERY Service: toolschecker: Redis set/get Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Thu Dec 12 06:25:02 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.020 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: wikilabels read/write is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: wikilabels read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:58 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/db/wikilabelsrw - 241 bytes in 16.022 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: expect a long running job on stretch is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: expect a long running job on stretch Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:57 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/portal:toolforge/admin/toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/continuous/stretch - 177 bytes in 0.028 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: start a job and verify on stretch is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: start a job and verify on stretch Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:57 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/portal:toolforge/admin/toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/stretch - 177 bytes in 0.085 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: toolsdb read/write is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: toolsdb read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:57 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/db/toolsdb - 241 bytes in 16.021 second response time

1 0

Cloud-admin-feed