Cloud-admin-feed October 2023

cloud-admin-feed@lists.wikimedia.org

2 participants
845 discussions

[FIRING:1] WidespreadPuppetAgentFailure cloudinfra (crit wmcs)
by root＠wmflabs.org 17 Oct '23

17 Oct '23

1 0

** PROBLEM alert - cloudvirt1051/ensure kvm processes are running is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Oct '23

17 Oct '23

Notification Type: PROBLEM Service: ensure kvm processes are running Host: cloudvirt1051 Address: 10.64.148.9 State: CRITICAL Date/Time: Tue Oct 17 15:46:46 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64

1 0

[FIRING:5] InstanceDown tools (node warn wmcs)
by root＠wmflabs.org 17 Oct '23

17 Oct '23

1 0

RECOVERY Host cloudvirt1051 - PING OK - Packet loss = 0%, RTA = 8.48 ms
by nagios＠alert1001.wikimedia.org 17 Oct '23

17 Oct '23

Notification Type: RECOVERY Host: cloudvirt1051 State: UP Address: 10.64.148.9 Info: PING OK - Packet loss = 0%, RTA = 8.48 ms Date/Time: Tue Oct 17 15:44:46 UTC 2023 Acknowledged by :

1 0

[RESOLVED] ProbeDown tools (172.16.4.136 ip4 tools-k8s-haproxy-4:30000 probes/custom warning prometheus wmcs)
by root＠wmflabs.org 17 Oct '23

17 Oct '23

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Oct '23

17 Oct '23

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Tue Oct 17 15:25:52 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.114 second response time

1 0

[FIRING:6] InstanceDown tools (node warn wmcs)
by root＠wmflabs.org 17 Oct '23

17 Oct '23

1 0

[FIRING:2] InstanceDown cloudinfra (node warn wmcs)
by root＠wmflabs.org 17 Oct '23

17 Oct '23

1 0

[FIRING:2] ProbeDown tools (172.16.4.136 ip4 tools-k8s-haproxy-4:30000 probes/custom warning prometheus wmcs)
by root＠wmflabs.org 17 Oct '23

17 Oct '23

1 0

PROBLEM Host cloudvirt1051 - PING CRITICAL - Packet loss = 100%
by nagios＠alert1001.wikimedia.org 17 Oct '23

17 Oct '23

Notification Type: PROBLEM Host: cloudvirt1051 State: DOWN Address: 10.64.148.9 Info: PING CRITICAL - Packet loss = 100% Date/Time: Tue Oct 17 15:16:30 UTC 2023 Acknowledged by :

1 0

← Newer
1
...
31
32
33
34
35
36
37
...
85
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed October 2023