September 2021 - Cloud-admin-feed

[FIRING:1] PuppetAgentDisabled tools (tools-package-builder-04 node warn)
by root＠wmflabs.org 27 Sep '21

27 Sep '21

1 0

** PROBLEM alert - cloudcontrol1004/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 27 Sep '21

27 Sep '21

Notification Type: PROBLEM Service: Galera haproxy failover Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Mon Sept 27 09:39:54 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: CRITICAL check_failover servers up 34 down 1

1 0

** RECOVERY alert - cloudcontrol1004/WMCS Galera Cluster is OK **
by nagios＠alert1001.wikimedia.org 27 Sep '21

27 Sep '21

Notification Type: RECOVERY Service: WMCS Galera Cluster Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Mon Sept 27 09:31:40 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: OK wsrep_cluster_size: 3, wsrep_cluster_status: Primary

1 0

** RECOVERY alert - cloudcontrol1004/WMCS Galera Database is OK **
by nagios＠alert1001.wikimedia.org 27 Sep '21

27 Sep '21

Notification Type: RECOVERY Service: WMCS Galera Database Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Mon Sept 27 09:31:16 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: Database seems up and running...

1 0

** RECOVERY alert - cloudcontrol1004/mysql -galera- process is OK **
by nagios＠alert1001.wikimedia.org 27 Sep '21

27 Sep '21

Notification Type: RECOVERY Service: mysql -galera- process Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Mon Sept 27 09:29:52 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS OK: 1 process with command name mysqld

1 0

** PROBLEM alert - cloudcontrol1004/Galera haproxy failover is CRITICAL **
by nagios＠alert1001.wikimedia.org 26 Sep '21

26 Sep '21

Notification Type: PROBLEM Service: Galera haproxy failover Host: cloudcontrol1004 Address: 208.80.154.132 State: CRITICAL Date/Time: Mon Sept 27 02:08:08 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: CRITICAL check_failover servers up 34 down 1

1 0

** RECOVERY alert - cloudcontrol1004/Galera haproxy failover is OK **
by nagios＠alert1001.wikimedia.org 26 Sep '21

26 Sep '21

Notification Type: RECOVERY Service: Galera haproxy failover Host: cloudcontrol1004 Address: 208.80.154.132 State: OK Date/Time: Mon Sept 27 02:02:22 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/HAProxy Acknowledged by : Additional Info: OK check_failover servers up 0 down 0

1 0

[FIRING:1] InstanceDown codesearch (codesearch8 node warn)
by root＠wmflabs.org 25 Sep '21

25 Sep '21

1 0

[FIRING:1] WidespreadInstanceDown codesearch (crit)
by root＠wmflabs.org 25 Sep '21

25 Sep '21

1 0

** RECOVERY alert - labstore1004/Persistent high iowait is OK **
by nagios＠alert1001.wikimedia.org 24 Sep '21

24 Sep '21

Notification Type: RECOVERY Service: Persistent high iowait Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Sat Sept 25 02:40:42 UTC 2021 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: (C)10 ge (W)5 ge 0.4211

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed September 2021