February 2022 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - cloudservices1004/Check unit status of labs-ip-alias-dump is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Feb '22

17 Feb '22

Notification Type: PROBLEM Service: Check unit status of labs-ip-alias-dump Host: cloudservices1004 Address: 208.80.154.11 State: CRITICAL Date/Time: Fri Feb 18 00:36:38 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time… Acknowledged by : Additional Info: CRITICAL: Status of the systemd unit labs-ip-alias-dump

1 0

** PROBLEM alert - cloudservices1004/Check systemd state is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Feb '22

17 Feb '22

Notification Type: PROBLEM Service: Check systemd state Host: cloudservices1004 Address: 208.80.154.11 State: CRITICAL Date/Time: Fri Feb 18 00:35:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The following units failed: labs-ip-alias-dump.service

1 0

** PROBLEM alert - cloudservices1003/Check unit status of labs-ip-alias-dump is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Feb '22

17 Feb '22

Notification Type: PROBLEM Service: Check unit status of labs-ip-alias-dump Host: cloudservices1003 Address: 208.80.154.135 State: CRITICAL Date/Time: Fri Feb 18 00:35:30 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time… Acknowledged by : Additional Info: CRITICAL: Status of the systemd unit labs-ip-alias-dump

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1005/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Feb '22

17 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: Galera haproxy failover Host: cloudcontrol1005 Address: 208.80.154.85 State: CRITICAL Date/Time: Thu Feb 17 23:07:15 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by andrew bogott: upgrading cloudcontrol1005 for T281276 Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1004/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Feb '22

17 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: Galera haproxy failover Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Thu Feb 17 23:07:14 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by andrew bogott: upgrading cloudcontrol1005 for T281276 Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

** ACKNOWLEDGEMENT alert - cloudcontrol1003/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Feb '22

17 Feb '22

Notification Type: ACKNOWLEDGEMENT Service: Galera haproxy failover Host: cloudcontrol1003 Address: 208.80.154.23 State: CRITICAL Date/Time: Thu Feb 17 23:07:14 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by andrew bogott: upgrading cloudcontrol1005 for T281276 Additional Info: CRITICAL check_failover servers up 34 down 1:

1 0

** PROBLEM alert - cloudvirt1017.mgmt/SSH is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Feb '22

17 Feb '22

Notification Type: PROBLEM Service: SSH Host: cloudvirt1017.mgmt Address: 10.65.5.68 State: CRITICAL Date/Time: Thu Feb 17 17:35:59 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_… Acknowledged by : Additional Info: CRITICAL - Socket timeout after 10 seconds

1 0

** RECOVERY alert - cloudvirt1017.mgmt/SSH is OK **
by nagios＠alert1001.wikimedia.org 17 Feb '22

17 Feb '22

Notification Type: RECOVERY Service: SSH Host: cloudvirt1017.mgmt Address: 10.65.5.68 State: OK Date/Time: Thu Feb 17 07:25:14 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_… Acknowledged by : Additional Info: SSH OK - OpenSSH_7.0 (protocol 2.0)

1 0

** PROBLEM alert - cloudvirt1017.mgmt/SSH is CRITICAL **
by nagios＠alert1001.wikimedia.org 16 Feb '22

16 Feb '22

Notification Type: PROBLEM Service: SSH Host: cloudvirt1017.mgmt Address: 10.65.5.68 State: CRITICAL Date/Time: Thu Feb 17 01:17:12 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_… Acknowledged by : Additional Info: CRITICAL - Socket timeout after 10 seconds

1 0

[FIRING:1] CloudinfraMariaDBWritableState cloudinfra (crit)
by root＠wmflabs.org 16 Feb '22

16 Feb '22

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed February 2022