June 2023 - Cloud-admin-feed - lists.wikimedia.org

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: check mtime mod from tools cron job is OK **
by nagios＠alert1001.wikimedia.org 08 Jun '23

08 Jun '23

Notification Type: RECOVERY Service: toolschecker: check mtime mod from tools cron job Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Thu Jun 8 13:25:57 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 11.866 second response time

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: NFS read/writeable on labs instances is OK **
by nagios＠alert1001.wikimedia.org 08 Jun '23

08 Jun '23

Notification Type: RECOVERY Service: toolschecker: NFS read/writeable on labs instances Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Thu Jun 8 13:25:51 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 28.579 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: check mtime mod from tools cron job is CRITICAL **
by nagios＠alert1001.wikimedia.org 08 Jun '23

08 Jun '23

Notification Type: PROBLEM Service: toolschecker: check mtime mod from tools cron job Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Jun 8 13:25:15 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/cron - 340 bytes in 60.012 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: NFS read/writeable on labs instances is CRITICAL **
by nagios＠alert1001.wikimedia.org 08 Jun '23

08 Jun '23

Notification Type: PROBLEM Service: toolschecker: NFS read/writeable on labs instances Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Jun 8 13:24:53 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/nfs/home - 340 bytes in 60.005 second response time

1 0

[FIRING:1] ProbeDown tools (172.16.2.173 ip4 tools-static-14:80 probes/custom http_tools_static_wmflabs_org_ip4 warning prometheus wmcs)
by root＠wmflabs.org 08 Jun '23

08 Jun '23

1 0

RECOVERY Host cloudcephosd1021 - PING OK - Packet loss = 0%, RTA = 0.21 ms
by nagios＠alert1001.wikimedia.org 08 Jun '23

08 Jun '23

Notification Type: RECOVERY Host: cloudcephosd1021 State: UP Address: 10.64.20.18 Info: PING OK - Packet loss = 0%, RTA = 0.21 ms Date/Time: Thu Jun 8 13:23:23 UTC 2023 Acknowledged by :

1 0

PROBLEM Host cloudcephosd1021 - PING CRITICAL - Packet loss = 100%
by nagios＠alert1001.wikimedia.org 08 Jun '23

08 Jun '23

Notification Type: PROBLEM Host: cloudcephosd1021 State: DOWN Address: 10.64.20.18 Info: PING CRITICAL - Packet loss = 100% Date/Time: Thu Jun 8 13:21:23 UTC 2023 Acknowledged by :

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s etcd nodes are healthy is CRITICAL **
by nagios＠alert1001.wikimedia.org 08 Jun '23

08 Jun '23

Notification Type: PROBLEM Service: toolschecker: All k8s etcd nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Jun 8 12:14:25 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/k8s - 516 bytes in 3.155 second response time

1 0

[RESOLVED] InstanceDown tools (tools-k8s-etcd-18 node warn wmcs)
by root＠wmflabs.org 08 Jun '23

08 Jun '23

1 0

[FIRING:1] InstanceDown tools (tools-k8s-etcd-18 node warn wmcs)
by root＠wmflabs.org 08 Jun '23

08 Jun '23

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed June 2023