September 2019 - Cloud-admin-feed

** RECOVERY alert - cloudcontrol1004/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: RECOVERY Service: Check systemd state Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Mon Sept 30 16:17:42 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - cloudcontrol1004/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: PROBLEM Service: Check systemd state Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Mon Sept 30 16:14:26 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: wikilabels read/write is OK **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: RECOVERY Service: toolschecker: wikilabels read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Mon Sept 30 10:29:55 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 51.018 second response time

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: toolsdb read/write is OK **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: RECOVERY Service: toolschecker: toolsdb read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Mon Sept 30 10:25:25 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.036 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: wikilabels read/write is CRITICAL **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: wikilabels read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Sept 30 09:39:26 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: updating ferm Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/db/wikilabelsrw - 340 bytes in 60.003 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: toolsdb read/write is CRITICAL **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: toolsdb read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Sept 30 09:39:25 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: updating ferm Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/db/toolsdb - 241 bytes in 10.016 second response time

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: Redis set/get is CRITICAL **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: Redis set/get Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Sept 30 09:39:25 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: updating ferm Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/redis - 340 bytes in 60.006 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: wikilabels read/write is CRITICAL **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: PROBLEM Service: toolschecker: wikilabels read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Sept 30 09:38:11 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/db/wikilabelsrw - 340 bytes in 60.003 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: toolsdb read/write is CRITICAL **
by nagios＠icinga1001.wikimedia.org 30 Sep '19

30 Sep '19

Notification Type: PROBLEM Service: toolschecker: toolsdb read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Sept 30 09:37:21 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/db/toolsdb - 241 bytes in 10.015 second response time

1 0

** RECOVERY alert - labstore1006/High 1m load average is OK **
by nagios＠icinga1001.wikimedia.org 29 Sep '19

29 Sep '19

Notification Type: RECOVERY Service: High 1m load average Host: labstore1006 Address: 208.80.154.7 State: OK Date/Time: Sun Sept 29 12:36:27 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: All metrics within thresholds.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed September 2019