February 2022 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - cloudcontrol1004/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: PROBLEM Service: Galera haproxy failover Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Sat Feb 19 06:30:03 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

** PROBLEM alert - cloudcontrol1004/WMCS Galera Cluster is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: PROBLEM Service: WMCS Galera Cluster Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Sat Feb 19 06:29:47 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: Caution: wsrep_cluster_status: Disconnected

1 0

** RECOVERY alert - cloudvirt1017.mgmt/SSH is OK **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: RECOVERY Service: SSH Host: cloudvirt1017.mgmt Address: 10.65.5.68 State: OK Date/Time: Sat Feb 19 01:41:21 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_… Acknowledged by : Additional Info: SSH OK - OpenSSH_7.0 (protocol 2.0)

1 0

** PROBLEM alert - cloudvirt1017.mgmt/SSH is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: PROBLEM Service: SSH Host: cloudvirt1017.mgmt Address: 10.65.5.68 State: CRITICAL Date/Time: Fri Feb 18 23:39:49 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_… Acknowledged by : Additional Info: CRITICAL - Socket timeout after 10 seconds

1 0

** RECOVERY alert - cloudcontrol1005/Galera haproxy failover is OK **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: RECOVERY Service: Galera haproxy failover Host: cloudcontrol1005 Address: 208.80.154.85 State: OK Date/Time: Fri Feb 18 23:37:17 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: OK check_failover servers up 35 down 0:

1 0

** RECOVERY alert - cloudcontrol1004/WMCS Galera Cluster is OK **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: RECOVERY Service: WMCS Galera Cluster Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Fri Feb 18 23:35:27 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: OK wsrep_cluster_size: 3, wsrep_cluster_status: Primary

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1005/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: Galera haproxy failover Host: cloudcontrol1005 Address: 208.80.154.85 State: CRITICAL Date/Time: Fri Feb 18 23:30:28 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by andrew bogott: investigating Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1004/WMCS Galera Cluster is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: WMCS Galera Cluster Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Fri Feb 18 23:30:27 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by andrew bogott: investigating Additional Info: Caution: wsrep_cluster_status: Disconnected

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1004/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: Galera haproxy failover Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Fri Feb 18 23:30:27 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by andrew bogott: investigating Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

** PROBLEM alert - cloudcontrol1005/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: PROBLEM Service: Galera haproxy failover Host: cloudcontrol1005 Address: 208.80.154.85 State: CRITICAL Date/Time: Fri Feb 18 23:23:16 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed February 2022