February 2022 - Cloud-admin-feed - lists.wikimedia.org

[FIRING:1] PuppetAgentFailure quarry (quarry-redis-01 node warn)
by root＠wmflabs.org 19 Feb '22

19 Feb '22

1 0

[FIRING:1] WidespreadPuppetAgentFailure quarry (crit)
by root＠wmflabs.org 19 Feb '22

19 Feb '22

1 0

[FIRING:1] InstanceDown quarry (quarry-nfs-1 node warn)
by root＠wmflabs.org 19 Feb '22

19 Feb '22

1 0

** RECOVERY alert - cloudcontrol1005/Galera haproxy failover is OK **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: RECOVERY Service: Galera haproxy failover Host: cloudcontrol1005 Address: 208.80.154.85 State: OK Date/Time: Sat Feb 19 06:44:47 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: OK check_failover servers up 35 down 0:

1 0

** RECOVERY alert - cloudcontrol1004/Galera haproxy failover is OK **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: RECOVERY Service: Galera haproxy failover Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Sat Feb 19 06:44:31 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: OK check_failover servers up 34 down 0:

1 0

** RECOVERY alert - cloudcontrol1004/WMCS Galera Cluster is OK **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: RECOVERY Service: WMCS Galera Cluster Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Sat Feb 19 06:44:17 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: OK wsrep_cluster_size: 3, wsrep_cluster_status: Primary

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1005/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: Galera haproxy failover Host: cloudcontrol1005 Address: 208.80.154.85 State: CRITICAL Date/Time: Sat Feb 19 06:40:26 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by andrew bogott: this seems to be flapping Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1004/WMCS Galera Cluster is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: WMCS Galera Cluster Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Sat Feb 19 06:40:26 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by andrew bogott: this seems to be flapping Additional Info: Caution: wsrep_cluster_status: Disconnected

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1004/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: Galera haproxy failover Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Sat Feb 19 06:40:26 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by andrew bogott: this seems to be flapping Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

** PROBLEM alert - cloudcontrol1005/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 18 Feb '22

18 Feb '22

Notification Type: PROBLEM Service: Galera haproxy failover Host: cloudcontrol1005 Address: 208.80.154.85 State: CRITICAL Date/Time: Sat Feb 19 06:30:19 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed February 2022