October 2019 - Cloud-admin-feed - lists.wikimedia.org

Host DOWN alert for cloudvirt1019!
by nagios＠icinga1001.wikimedia.org 24 Oct '19

24 Oct '19

Notification Type: PROBLEM Host: cloudvirt1019 State: DOWN Address: 10.64.20.38 Info: PING CRITICAL - Packet loss = 100% Date/Time: Thu Oct 24 11:56:06 UTC 2019 Acknowledged by :

1 0

** ACKNOWLEDGEMENT alert - checker.tools.wmflabs.org/toolschecker: toolsdb read/write is CRITICAL **
by nagios＠icinga1001.wikimedia.org 24 Oct '19

24 Oct '19

Notification Type: ACKNOWLEDGEMENT Service: toolschecker: toolsdb read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Oct 24 11:10:39 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by Arturo Borrero Gonzalez: PDU ops Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/db/toolsdb - 241 bytes in 0.008 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: toolsdb read/write is CRITICAL **
by nagios＠icinga1001.wikimedia.org 24 Oct '19

24 Oct '19

Notification Type: PROBLEM Service: toolschecker: toolsdb read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Oct 24 11:07:25 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/db/toolsdb - 241 bytes in 0.016 second response time

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1018/MegaRAID is CRITICAL **
by nagios＠icinga1001.wikimedia.org 24 Oct '19

24 Oct '19

Notification Type: ACKNOWLEDGEMENT Service: MegaRAID Host: cloudvirt1018 Address: 10.64.20.34 State: CRITICAL Date/Time: Wed Oct 23 23:36:44 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Acknowledged by nagiosadmin: RAID handler auto-ack: https://phabricator.wikimedia.org/T236331 Additional Info: CRITICAL: 1 failed LD(s) (Degraded)

1 0

** PROBLEM alert - cloudvirt1018/MegaRAID is CRITICAL **
by nagios＠icinga1001.wikimedia.org 24 Oct '19

24 Oct '19

Notification Type: PROBLEM Service: MegaRAID Host: cloudvirt1018 Address: 10.64.20.34 State: CRITICAL Date/Time: Wed Oct 23 23:36:41 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Acknowledged by : Additional Info: CRITICAL: 1 failed LD(s) (Degraded)

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is OK **
by nagios＠icinga1001.wikimedia.org 23 Oct '19

23 Oct '19

Notification Type: RECOVERY Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Tue Oct 22 17:00:26 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.130 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 23 Oct '19

23 Oct '19

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Tue Oct 22 16:49:14 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.142 second response time

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: check mtime mod from tools cron job is OK **
by nagios＠icinga1001.wikimedia.org 23 Oct '19

23 Oct '19

Notification Type: RECOVERY Service: toolschecker: check mtime mod from tools cron job Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Tue Oct 22 14:14:12 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.004 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: check mtime mod from tools cron job is CRITICAL **
by nagios＠icinga1001.wikimedia.org 23 Oct '19

23 Oct '19

Notification Type: PROBLEM Service: toolschecker: check mtime mod from tools cron job Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Tue Oct 22 14:09:24 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/cron - 177 bytes in 0.008 second response time

1 0

** ACKNOWLEDGEMENT alert - cloudbackup2001/Check the last execution of block_sync-tools-project is CRITICAL **
by nagios＠icinga1001.wikimedia.org 18 Oct '19

18 Oct '19

Notification Type: ACKNOWLEDGEMENT Service: Check the last execution of block_sync-tools-project Host: cloudbackup2001 Address: 10.192.0.130 State: CRITICAL Date/Time: Thu Oct 17 14:53:45 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time… Acknowledged by andrew bogott: this will run again tonight, well see if it works better. Additional Info: CRITICAL: Status of the systemd unit block_sync-tools-project

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed October 2019