October 2022 - Cloud-admin-feed - lists.wikimedia.org

[FIRING:1] PuppetAgentNoResources project-proxy (proxy-03 node warn)
by root＠wmflabs.org 24 Oct '22

24 Oct '22

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is OK **
by nagios＠alert1001.wikimedia.org 24 Oct '22

24 Oct '22

Notification Type: RECOVERY Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Mon Oct 24 16:41:15 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.099 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠alert1001.wikimedia.org 24 Oct '22

24 Oct '22

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Mon Oct 24 16:23:11 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.106 second response time

1 0

** RECOVERY alert - cloudcontrol1005/Check unit status of backup_cinder_volumes is OK **
by nagios＠alert1001.wikimedia.org 24 Oct '22

24 Oct '22

Notification Type: RECOVERY Service: Check unit status of backup_cinder_volumes Host: cloudcontrol1005 Address: 208.80.154.85 State: OK Date/Time: Mon Oct 24 10:33:52 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_u… Acknowledged by : Additional Info: OK: Status of the systemd unit backup_cinder_volumes

1 0

** RECOVERY alert - codesearch.wmcloud.org/HTTPS-cloud-vps-proxy is OK **
by nagios＠alert1001.wikimedia.org 22 Oct '22

22 Oct '22

Notification Type: RECOVERY Service: HTTPS-cloud-vps-proxy Host: codesearch.wmcloud.org Address: codesearch.wmcloud.org State: OK Date/Time: Sat Oct 22 12:16:55 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Acme-chief/Cloud_VPS_setup%23Troublesho… Acknowledged by : Additional Info: SSL OK - Certificate wmflabs.org valid until 2022-12-18 12:34:10 +0000 (expires in 57 days)

1 0

** PROBLEM alert - codesearch.wmcloud.org/HTTPS-cloud-vps-proxy is CRITICAL **
by nagios＠alert1001.wikimedia.org 22 Oct '22

22 Oct '22

Notification Type: PROBLEM Service: HTTPS-cloud-vps-proxy Host: codesearch.wmcloud.org Address: codesearch.wmcloud.org State: CRITICAL Date/Time: Sat Oct 22 12:14:49 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Acme-chief/Cloud_VPS_setup%23Troublesho… Acknowledged by : Additional Info: SSL CRITICAL - failed to connect or SSL handshake:Connection reset by peer

1 0

** PROBLEM alert - cloudcontrol1005/Check unit status of backup_cinder_volumes is CRITICAL **
by nagios＠alert1001.wikimedia.org 21 Oct '22

21 Oct '22

Notification Type: PROBLEM Service: Check unit status of backup_cinder_volumes Host: cloudcontrol1005 Address: 208.80.154.85 State: CRITICAL Date/Time: Fri Oct 21 16:35:03 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_u… Acknowledged by : Additional Info: CRITICAL: Status of the systemd unit backup_cinder_volumes

1 0

[FIRING:1] InstanceDown tools (tools-k8s-worker-62 node warn)
by root＠wmflabs.org 21 Oct '22

21 Oct '22

1 0

** RECOVERY alert - cloudcontrol1005/Check unit status of backup_cinder_volumes is OK **
by nagios＠alert1001.wikimedia.org 21 Oct '22

21 Oct '22

Notification Type: RECOVERY Service: Check unit status of backup_cinder_volumes Host: cloudcontrol1005 Address: 208.80.154.85 State: OK Date/Time: Fri Oct 21 10:40:16 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_u… Acknowledged by : Additional Info: OK: Status of the systemd unit backup_cinder_volumes

1 0

** RECOVERY alert - cloudrabbit1001/Check systemd state is OK **
by nagios＠alert1001.wikimedia.org 21 Oct '22

21 Oct '22

Notification Type: RECOVERY Service: Check systemd state Host: cloudrabbit1001 Address: 208.80.154.147 State: OK Date/Time: Fri Oct 21 07:36:18 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed October 2022