December 2019 - Cloud-admin-feed - lists.wikimedia.org

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: start a job and verify on stretch is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: start a job and verify on stretch Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:57 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/portal:toolforge/admin/toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/stretch - 177 bytes in 0.085 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: expect a long running job on stretch is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: expect a long running job on stretch Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:57 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/portal:toolforge/admin/toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/continuous/stretch - 177 bytes in 0.028 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: Redis set/get is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: Redis set/get Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:56 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/redis - 340 bytes in 60.006 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:56 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 516 bytes in 16.021 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: All k8s etcd nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: All k8s etcd nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:56 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/k8s - 519 bytes in 30.276 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: All Flannel etcd nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: All Flannel etcd nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:21:56 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: upgrading openstack Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/flannel - 523 bytes in 16.022 second response time

1 0

** RECOVERY alert - labstore1005/Ensure NFS exports are maintained for new instances with NFS is OK **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: RECOVERY Service: Ensure NFS exports are maintained for new instances with NFS Host: labstore1005 Address: 10.64.37.20 State: OK Date/Time: Thu Dec 12 06:20:22 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Acknowledged by : Additional Info: OK - nfs-exportd is active

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: Redis set/get is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: PROBLEM Service: toolschecker: Redis set/get Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Dec 12 06:18:54 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/redis - 340 bytes in 60.009 second response time

1 0

** ACKNOWLEDGEMENT alert - cloudstore1008/Ensure NFS exports are maintained for new instances with NFS is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: Ensure NFS exports are maintained for new instances with NFS Host: cloudstore1008 Address: 208.80.155.125 State: CRITICAL Date/Time: Thu Dec 12 06:13:15 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Acknowledged by andrew bogott: side-effect of ocata upgrade Additional Info: CRITICAL - Expecting active but unit nfs-exportd is activating

1 0

** ACKNOWLEDGEMENT alert - cloudstore1009/Ensure NFS exports are maintained for new instances with NFS is CRITICAL **
by nagios＠icinga1001.wikimedia.org 12 Dec '19

12 Dec '19

Notification Type: ACKNOWLEDGEMENT Service: Ensure NFS exports are maintained for new instances with NFS Host: cloudstore1009 Address: 208.80.155.126 State: CRITICAL Date/Time: Thu Dec 12 06:12:00 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Acknowledged by andrew bogott: side-effect of upgrade in progress Additional Info: CRITICAL - Expecting active but unit nfs-exportd is activating

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed December 2019