Notification Type: PROBLEM
Service: Check for VMs leaked by the nova-fullstack test
Host: cloudcontrol1003
Address: 208.80.154.23
State: CRITICAL
Date/Time: Wed Feb 23 10:00:37 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_f…
Acknowledged by :
Additional Info:
10 instances in the admin-monitoring project
Notification Type: PROBLEM
Service: Check for VMs leaked by the nova-fullstack test
Host: cloudcontrol1003
Address: 208.80.154.23
State: CRITICAL
Date/Time: Wed Feb 23 08:08:31 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Check_f…
Acknowledged by :
Additional Info:
10 instances in the admin-monitoring project
Notification Type: RECOVERY
Service: Galera haproxy failover
Host: cloudcontrol1003
Address: 208.80.154.23
State: OK
Date/Time: Wed Feb 23 06:03:16 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy
Acknowledged by :
Additional Info:
OK check_failover servers up 32 down 0:
Notification Type: RECOVERY
Service: Galera haproxy failover
Host: cloudcontrol1005
Address: 208.80.154.85
State: OK
Date/Time: Wed Feb 23 06:02:40 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy
Acknowledged by :
Additional Info:
OK check_failover servers up 32 down 0:
Notification Type: PROBLEM
Service: WMCS Galera Database
Host: cloudcontrol1003
Address: 208.80.154.23
State: CRITICAL
Date/Time: Wed Feb 23 05:59:30 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
Acknowledged by :
Additional Info:
Error during connection: Cant connect to MySQL server on 208.80.154.23 (115)
Notification Type: RECOVERY
Host: cloudcontrol1004
State: UP
Address: 208.80.154.132
Info: PING OK - Packet loss = 0%, RTA = 0.25 ms
Date/Time: Wed Feb 23 05:39:23 UTC 2022
Acknowledged by :
Notification Type: PROBLEM
Host: cloudcontrol1004
State: DOWN
Address: 208.80.154.132
Info: PING CRITICAL - Packet loss = 100%
Date/Time: Wed Feb 23 05:35:59 UTC 2022
Acknowledged by :
Notification Type: ACKNOWLEDGEMENT
Service: WMCS Galera Database
Host: cloudcontrol1004
Address: 208.80.154.132
State: CRITICAL
Date/Time: Wed Feb 23 05:02:02 UTC 2022
Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting
Acknowledged by andrew bogott: T302146
Additional Info:
Error during connection: Cant connect to MySQL server on 208.80.154.132 (115)