Notification Type: PROBLEM
Service: WMCS Galera Database
Host: cloudcontrol1004
Address: 208.80.154.132
State: CRITICAL
Date/Time: Wed Feb 23 05:00:18 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
Acknowledged by :
Additional Info:
Error during connection: Cant connect to MySQL server on 208.80.154.132 (115)
Notification Type: ACKNOWLEDGEMENT
Service: Check unit status of backup_cinder_volumes
Host: cloudcontrol1005
Address: 208.80.154.85
State: CRITICAL
Date/Time: Wed Feb 23 01:41:24 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time…
Acknowledged by andrew bogott: this should recover if galera settles down
Additional Info:
CRITICAL: Status of the systemd unit backup_cinder_volumes
Notification Type: RECOVERY
Service: Galera haproxy failover
Host: cloudcontrol1003
Address: 208.80.154.23
State: OK
Date/Time: Wed Feb 23 01:36:21 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy
Acknowledged by :
Additional Info:
OK check_failover servers up 32 down 0:
Notification Type: RECOVERY
Service: Galera haproxy failover
Host: cloudcontrol1005
Address: 208.80.154.85
State: OK
Date/Time: Wed Feb 23 00:53:51 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy
Acknowledged by :
Additional Info:
OK check_failover servers up 32 down 0: