April 2021 - Cloud-admin-feed - lists.wikimedia.org

[FIRING:1] PuppetAgentFailures tools (tools-k8s-etcd-8 node warn)
by root＠wmflabs.org 21 Apr '21

21 Apr '21

1 0

[FIRING:1] InstanceDown tools (tools-sgeexec-0919 node warn)
by root＠wmflabs.org 20 Apr '21

20 Apr '21

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s etcd nodes are healthy is OK **
by nagios＠alert1001.wikimedia.org 20 Apr '21

20 Apr '21

Notification Type: RECOVERY Service: toolschecker: All k8s etcd nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Tue Apr 20 16:20:33 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.500 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s etcd nodes are healthy is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Apr '21

20 Apr '21

Notification Type: PROBLEM Service: toolschecker: All k8s etcd nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Tue Apr 20 16:14:25 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/etcd/k8s - 177 bytes in 3.640 second response time

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1045/ensure kvm processes are running is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Apr '21

20 Apr '21

Notification Type: ACKNOWLEDGEMENT Service: ensure kvm processes are running Host: cloudvirt1045 Address: 10.64.20.11 State: CRITICAL Date/Time: Tue Apr 20 15:33:00 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by andrew bogott: setup in progress Additional Info: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1040/Check systemd state is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Apr '21

20 Apr '21

Notification Type: ACKNOWLEDGEMENT Service: Check systemd state Host: cloudvirt1040 Address: 10.64.20.6 State: CRITICAL Date/Time: Tue Apr 20 15:33:00 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by andrew bogott: setup in progress Additional Info: CRITICAL - degraded: The following units failed: nova-compute.service

1 0

** PROBLEM alert - cloudvirt1045/ensure kvm processes are running is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Apr '21

20 Apr '21

Notification Type: PROBLEM Service: ensure kvm processes are running Host: cloudvirt1045 Address: 10.64.20.11 State: CRITICAL Date/Time: Tue Apr 20 15:26:16 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64

1 0

** RECOVERY alert - cloudvirt1018/MegaRAID is OK **
by nagios＠alert1001.wikimedia.org 20 Apr '21

20 Apr '21

Notification Type: RECOVERY Service: MegaRAID Host: cloudvirt1018 Address: 10.64.20.34 State: OK Date/Time: Tue Apr 20 14:27:38 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Acknowledged by : Additional Info: OK: optimal, 1 logical, 8 physical

1 0

** ACKNOWLEDGEMENT alert - cloudvirt1018/MegaRAID is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Apr '21

20 Apr '21

Notification Type: ACKNOWLEDGEMENT Service: MegaRAID Host: cloudvirt1018 Address: 10.64.20.34 State: CRITICAL Date/Time: Tue Apr 20 12:41:42 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Acknowledged by nagiosadmin: RAID handler auto-ack: https://phabricator.wikimedia.org/T280668 Additional Info: CRITICAL: 1 failed LD(s) (Degraded)

1 0

** PROBLEM alert - cloudvirt1018/MegaRAID is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Apr '21

20 Apr '21

Notification Type: PROBLEM Service: MegaRAID Host: cloudvirt1018 Address: 10.64.20.34 State: CRITICAL Date/Time: Tue Apr 20 12:41:41 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring Acknowledged by : Additional Info: CRITICAL: 1 failed LD(s) (Degraded)

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed April 2021