January 2020 - Cloud-admin-feed - lists.wikimedia.org

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: All k8s worker nodes are healthy is CRITICAL **
by nagios＠icinga1001.wikimedia.org 22 Jan '20

22 Jan '20

Notification Type: PROBLEM Service: toolschecker: All k8s worker nodes are healthy Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Wed Jan 22 02:14:19 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 177 bytes in 0.214 second response time

1 0

** RECOVERY alert - labstore2003/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 21 Jan '20

21 Jan '20

Notification Type: RECOVERY Service: Check systemd state Host: labstore2003 Address: 10.192.21.6 State: OK Date/Time: Tue Jan 21 18:55:45 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - labstore2003/Check systemd state is CRITICAL **
by nagios＠icinga1001.wikimedia.org 21 Jan '20

21 Jan '20

Notification Type: PROBLEM Service: Check systemd state Host: labstore2003 Address: 10.192.21.6 State: CRITICAL Date/Time: Tue Jan 21 18:53:55 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The system is operational but one or more units failed.

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: Make sure enwiki dumps are not empty is OK **
by nagios＠icinga1001.wikimedia.org 21 Jan '20

21 Jan '20

Notification Type: RECOVERY Service: toolschecker: Make sure enwiki dumps are not empty Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Tue Jan 21 17:23:21 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 1.866 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: Make sure enwiki dumps are not empty is CRITICAL **
by nagios＠icinga1001.wikimedia.org 21 Jan '20

21 Jan '20

Notification Type: PROBLEM Service: toolschecker: Make sure enwiki dumps are not empty Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Tue Jan 21 17:10:25 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker Acknowledged by : Additional Info: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/nfs/dumps - 258 bytes in 0.011 second response time

1 0

** RECOVERY alert - cloudcontrol1003/nova instance creation test is OK **
by nagios＠icinga1001.wikimedia.org 15 Jan '20

15 Jan '20

Notification Type: RECOVERY Service: nova instance creation test Host: cloudcontrol1003 Address: 208.80.154.23 State: OK Date/Time: Wed Jan 15 15:29:26 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS OK: 1 process with command name python, args nova-fullstack

1 0

** PROBLEM alert - cloudcontrol1003/nova instance creation test is CRITICAL **
by nagios＠icinga1001.wikimedia.org 15 Jan '20

15 Jan '20

Notification Type: PROBLEM Service: nova instance creation test Host: cloudcontrol1003 Address: 208.80.154.23 State: CRITICAL Date/Time: Wed Jan 15 15:15:29 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting Acknowledged by : Additional Info: PROCS CRITICAL: 0 processes with command name python, args nova-fullstack

1 0

** RECOVERY alert - labstore1004/Ensure mysql credential creation for tools users is running is OK **
by nagios＠icinga1001.wikimedia.org 15 Jan '20

15 Jan '20

Notification Type: RECOVERY Service: Ensure mysql credential creation for tools users is running Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Wed Jan 15 00:22:53 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state Acknowledged by : Additional Info: OK - maintain-dbusers is active

1 0

** RECOVERY alert - labstore1004/Check systemd state is OK **
by nagios＠icinga1001.wikimedia.org 15 Jan '20

15 Jan '20

Notification Type: RECOVERY Service: Check systemd state Host: labstore1004 Address: 10.64.37.19 State: OK Date/Time: Wed Jan 15 00:21:53 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** RECOVERY alert - labstore1007/High 1m load average is OK **
by nagios＠icinga1001.wikimedia.org 15 Jan '20

15 Jan '20

Notification Type: RECOVERY Service: High 1m load average Host: labstore1007 Address: 208.80.155.106 State: OK Date/Time: Wed Jan 15 00:15:29 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore https://grafana.wikimedia.org/dashboard/db/labs-monitoring Acknowledged by : Additional Info: All metrics within thresholds.

1 0

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed January 2020