December 2022 - Cloud-admin-feed - lists.wikimedia.org

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: toolsdb read/write is OK **
by nagios＠alert1001.wikimedia.org 07 Dec '22

07 Dec '22

Notification Type: RECOVERY Service: toolschecker: toolsdb read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Wed Dec 7 22:51:18 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.024 second response time

1 0

[FIRING:1] PuppetAgentNoResources clouddb-services (clouddb1003 node warn)
by root＠wmflabs.org 07 Dec '22

07 Dec '22

1 0

** RECOVERY alert - cloudvirt1019/ensure kvm processes are running is OK **
by nagios＠alert1001.wikimedia.org 07 Dec '22

07 Dec '22

Notification Type: RECOVERY Service: ensure kvm processes are running Host: cloudvirt1019 Address: 10.64.20.38 State: OK Date/Time: Wed Dec 7 22:11:41 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS OK: 5 processes with regex args qemu-system-x86_64

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: toolsdb read/write is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Dec '22

07 Dec '22

Notification Type: PROBLEM Service: toolschecker: toolsdb read/write Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Wed Dec 7 22:10:17 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/db/toolsdb - 241 bytes in 3.075 second response time

1 0

** PROBLEM alert - cloudvirt1019/ensure kvm processes are running is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Dec '22

07 Dec '22

Notification Type: PROBLEM Service: ensure kvm processes are running Host: cloudvirt1019 Address: 10.64.20.38 State: CRITICAL Date/Time: Wed Dec 7 22:09:11 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64

1 0

** RECOVERY alert - cloudbackup1004/Check unit status of backup_vms is OK **
by nagios＠alert1001.wikimedia.org 06 Dec '22

06 Dec '22

Notification Type: RECOVERY Service: Check unit status of backup_vms Host: cloudbackup1004 Address: 10.64.20.23 State: OK Date/Time: Tue Dec 6 17:19:04 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_u… Acknowledged by : Additional Info: OK: Status of the systemd unit backup_vms

1 0

** RECOVERY alert - labstore1004/NFS Share Volume Space /srv/tools is OK **
by nagios＠alert1001.wikimedia.org 06 Dec '22

06 Dec '22

Notification Type: RECOVERY Service: NFS Share Volume Space /srv/tools Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Tue Dec 6 11:03:06 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_stora… https://grafana.wikimedia.org/d/50z0i4XWz/tools-overall-nfs-storage-utiliza… Acknowledged by : Additional Info: DISK OK - free space: /srv/tools 1826019 MB (23% inode=67%):

1 0

[RESOLVED] InstanceDown tools (tools-redis-6 node warn)
by root＠wmflabs.org 05 Dec '22

05 Dec '22

1 0

** RECOVERY alert - cloudcontrol1005/Check unit status of backup_cinder_volumes is OK **
by nagios＠alert1001.wikimedia.org 05 Dec '22

05 Dec '22

Notification Type: RECOVERY Service: Check unit status of backup_cinder_volumes Host: cloudcontrol1005 Address: 208.80.154.85 State: OK Date/Time: Mon Dec 5 17:55:08 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_u… Acknowledged by : Additional Info: OK: Status of the systemd unit backup_cinder_volumes

1 0

[FIRING:1] InstanceDown tools (tools-redis-6 node warn)
by root＠wmflabs.org 05 Dec '22

05 Dec '22

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed December 2022