March 2023 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠alert1001.wikimedia.org 12 Mar '23

12 Mar '23

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Sun Mar 12 13:30:39 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 340 bytes in 60.022 second response time

1 0

[FIRING:2] ProbeDown tools (172.16.5.92 ip4 tools-k8s-haproxy-3:30000 probes/custom warning prometheus wmcs)
by root＠wmflabs.org 12 Mar '23

12 Mar '23

1 0

[FIRING:1] ProbeDown tools (172.16.5.92 ip4 tools-k8s-haproxy-3:30000 probes/custom http_admin_toolforge_org_ip4 warning prometheus wmcs)
by root＠wmflabs.org 12 Mar '23

12 Mar '23

1 0

[RESOLVED] ProbeDown tools (172.16.5.92 ip4 tools-k8s-haproxy-3:30000 probes/custom http_admin_toolforge_org_ip4 warning prometheus wmcs)
by root＠wmflabs.org 12 Mar '23

12 Mar '23

1 0

[FIRING:1] ProbeDown tools (172.16.5.92 ip4 tools-k8s-haproxy-3:30000 probes/custom warning prometheus wmcs)
by root＠wmflabs.org 12 Mar '23

12 Mar '23

1 0

[FIRING:2] ProbeDown tools (172.16.5.92 ip4 tools-k8s-haproxy-3:30000 probes/custom warning prometheus wmcs)
by root＠wmflabs.org 12 Mar '23

12 Mar '23

1 0

** PROBLEM alert - cloudbackup1003/Check unit status of backup_glance_images is CRITICAL **
by nagios＠alert1001.wikimedia.org 11 Mar '23

11 Mar '23

Notification Type: PROBLEM Service: Check unit status of backup_glance_images Host: cloudbackup1003 Address: 10.64.20.22 State: CRITICAL Date/Time: Sat Mar 11 13:03:39 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Acknowledged by : Additional Info: CRITICAL: Status of the systemd unit backup_glance_images

1 0

** RECOVERY alert - cloudcontrol1005/all nova flavors are assigned necessary properties is OK **
by nagios＠alert1001.wikimedia.org 11 Mar '23

11 Mar '23

Notification Type: RECOVERY Service: all nova flavors are assigned necessary properties Host: cloudcontrol1005 Address: 208.80.154.85 State: OK Date/Time: Sat Mar 11 12:53:07 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Host_aggregates Acknowledged by : Additional Info: All flavors are assigned to aggregates

1 0

** PROBLEM alert - cloudcontrol1005/all nova flavors are assigned necessary properties is CRITICAL **
by nagios＠alert1001.wikimedia.org 11 Mar '23

11 Mar '23

Notification Type: PROBLEM Service: all nova flavors are assigned necessary properties Host: cloudcontrol1005 Address: 208.80.154.85 State: CRITICAL Date/Time: Sat Mar 11 12:37:17 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Host_aggregates Acknowledged by : Additional Info: Some flavors are not assigned to aggregates: g2.cores8.ram16.disk1120.custom, g2.cores2.ram4.disk40.custom

1 0

** PROBLEM alert - cloudbackup1003/Check unit status of backup_vms is CRITICAL **
by nagios＠alert1001.wikimedia.org 11 Mar '23

11 Mar '23

Notification Type: PROBLEM Service: Check unit status of backup_vms Host: cloudbackup1003 Address: 10.64.20.22 State: CRITICAL Date/Time: Sat Mar 11 05:07:07 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_u… Acknowledged by : Additional Info: CRITICAL: Status of the systemd unit backup_vms

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed March 2023