September 2022 - Cloud-admin-feed

[FIRING:2] ProbeDown tools (ip4 probes/custom http_admin_toolforge_org_ip4 warning prometheus wmcs)
by root＠wmflabs.org 30 Sep '22

30 Sep '22

1 0

[FIRING:1] ProbeDown tools (172.16.4.136 ip4 tools-k8s-haproxy-4:30000 probes/custom http_admin_toolforge_org_ip4 warning prometheus wmcs)
by root＠wmflabs.org 30 Sep '22

30 Sep '22

1 0

[FIRING:1] ProbeDown tools (172.16.4.136 ip4 tools-k8s-haproxy-4:30000 probes/custom http_admin_toolforge_org_ip4 warning prometheus wmcs)
by root＠wmflabs.org 30 Sep '22

30 Sep '22

1 0

[FIRING:2] ProbeDown tools (ip4 probes/custom http_admin_toolforge_org_ip4 warning prometheus wmcs)
by root＠wmflabs.org 30 Sep '22

30 Sep '22

1 0

[FIRING:1] ProbeDown tools (172.16.5.92 ip4 tools-k8s-haproxy-3:30000 probes/custom http_admin_toolforge_org_ip4 warning prometheus wmcs)
by root＠wmflabs.org 30 Sep '22

30 Sep '22

1 0

** RECOVERY alert - cloudcontrol1005/Check unit status of backup_cinder_volumes is OK **
by nagios＠alert1001.wikimedia.org 30 Sep '22

30 Sep '22

Notification Type: RECOVERY Service: Check unit status of backup_cinder_volumes Host: cloudcontrol1005 Address: 208.80.154.85 State: OK Date/Time: Thu Sept 29 22:11:10 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_u… Acknowledged by : Additional Info: OK: Status of the systemd unit backup_cinder_volumes

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is OK **
by nagios＠alert1001.wikimedia.org 29 Sep '22

29 Sep '22

Notification Type: RECOVERY Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Thu Sept 29 19:54:39 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.125 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠alert1001.wikimedia.org 29 Sep '22

29 Sep '22

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Thu Sept 29 19:48:34 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 394 bytes in 3.040 second response time

1 0

[FIRING:1] InstanceDown tools (tools-k8s-control-1 node warn)
by root＠wmflabs.org 29 Sep '22

29 Sep '22

1 0

[FIRING:1] InstanceDown tools (tools-k8s-control-1 node warn)
by root＠wmflabs.org 29 Sep '22

29 Sep '22

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed September 2022