Notification Type: RECOVERY
Service: Check unit status of backup_glance_images
Host: cloudcontrol1005
Address: 208.80.154.85
State: OK
Date/Time: Mon Jan 31 10:18:18 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time…
Acknowledged by :
Additional Info:
OK: Status of the systemd unit backup_glance_images
Notification Type: PROBLEM
Service: Check unit status of backup_glance_images
Host: cloudcontrol1005
Address: 208.80.154.85
State: CRITICAL
Date/Time: Sun Jan 30 17:20:44 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time…
Acknowledged by :
Additional Info:
CRITICAL: Status of the systemd unit backup_glance_images
Notification Type: RECOVERY
Service: Check unit status of wmcs_monitoring_graphite_rsync
Host: cloudmetrics1004
Address: 10.64.37.6
State: OK
Date/Time: Thu Jan 27 13:32:55 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time…
Acknowledged by :
Additional Info:
OK: Status of the systemd unit wmcs_monitoring_graphite_rsync
Notification Type: RECOVERY
Service: Check systemd state
Host: cloudmetrics1004
Address: 10.64.37.6
State: OK
Date/Time: Thu Jan 27 13:24:15 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
Acknowledged by :
Additional Info:
OK - running: The system is fully operational
Notification Type: PROBLEM
Service: Check unit status of wmcs_monitoring_graphite_rsync
Host: cloudmetrics1004
Address: 10.64.37.6
State: CRITICAL
Date/Time: Thu Jan 27 13:10:25 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time…
Acknowledged by :
Additional Info:
CRITICAL: Status of the systemd unit wmcs_monitoring_graphite_rsync
Notification Type: PROBLEM
Service: Check systemd state
Host: cloudmetrics1004
Address: 10.64.37.6
State: CRITICAL
Date/Time: Thu Jan 27 13:06:39 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
Acknowledged by :
Additional Info:
CRITICAL - degraded: The following units failed: wmcs_monitoring_graphite_rsync.service
Notification Type: RECOVERY
Service: DPKG
Host: cloudcontrol1004
Address: 208.80.154.132
State: OK
Date/Time: Wed Jan 26 21:24:35 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/dpkg
Acknowledged by :
Additional Info:
All packages OK
Notification Type: RECOVERY
Service: Galera haproxy failover
Host: cloudcontrol1005
Address: 208.80.154.85
State: OK
Date/Time: Wed Jan 26 21:11:49 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy
Acknowledged by :
Additional Info:
OK check_failover servers up 35 down 0:
Notification Type: ACKNOWLEDGEMENT
Service: WMCS Galera Database
Host: cloudcontrol1005
Address: 208.80.154.85
State: CRITICAL
Date/Time: Wed Jan 26 21:11:45 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
Acknowledged by andrew bogott: I restarted this by accident
Additional Info:
Error during connection: Lost connection to MySQL server at handshake: reading initial communication packet, system error: 104