July 2018 - Cloud-admin-feed - lists.wikimedia.org

Host UP alert for labvirt1009!
by icinga＠einsteinium.wikimedia.org 31 Jul '18

31 Jul '18

Notification Type: RECOVERY Host: labvirt1009 State: UP Address: 10.64.20.15 Info: PING OK - Packet loss = 0%, RTA = 0.66 ms Date/Time: Tue Jul 31 17:30:51 UTC 2018

1 0

Host DOWN alert for labvirt1009!
by icinga＠einsteinium.wikimedia.org 31 Jul '18

31 Jul '18

Notification Type: PROBLEM Host: labvirt1009 State: DOWN Address: 10.64.20.15 Info: PING CRITICAL - Packet loss = 100% Date/Time: Tue Jul 31 17:26:58 UTC 2018

1 0

** RECOVERY alert - labvirt1010/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 28 Jul '18

28 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labvirt1010 Address: 10.64.20.26 State: OK Date/Time: Sat Jul 28 06:57:29 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures

1 0

** PROBLEM alert - labvirt1010/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 28 Jul '18

28 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labvirt1010 Address: 10.64.20.26 State: CRITICAL Date/Time: Sat Jul 28 06:31:49 UTC 2018 Notes URLs: Additional Info: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/share/ca-certificates/DigiCert_High_Assurance_CA-3.crt]

1 0

** RECOVERY alert - labstore1005/High load average is OK **
by icinga＠einsteinium.wikimedia.org 26 Jul '18

26 Jul '18

Notification Type: RECOVERY Service: High load average Host: labstore1005 Address: 10.64.37.20 State: OK Date/Time: Thu Jul 26 19:07:41 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: OK: Less than 50.00% above the threshold [16.0]

1 0

** PROBLEM alert - labstore1005/High load average is CRITICAL **
by icinga＠einsteinium.wikimedia.org 26 Jul '18

26 Jul '18

Notification Type: PROBLEM Service: High load average Host: labstore1005 Address: 10.64.37.20 State: CRITICAL Date/Time: Thu Jul 26 18:08:11 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: CRITICAL: 77.78% of data above the critical threshold [24.0]

1 0

** RECOVERY alert - labstore1005/High load average is OK **
by icinga＠einsteinium.wikimedia.org 26 Jul '18

26 Jul '18

Notification Type: RECOVERY Service: High load average Host: labstore1005 Address: 10.64.37.20 State: OK Date/Time: Thu Jul 26 17:56:41 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: OK: Less than 50.00% above the threshold [16.0]

1 0

** PROBLEM alert - labstore1005/High load average is CRITICAL **
by icinga＠einsteinium.wikimedia.org 26 Jul '18

26 Jul '18

Notification Type: PROBLEM Service: High load average Host: labstore1005 Address: 10.64.37.20 State: CRITICAL Date/Time: Thu Jul 26 17:52:11 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: CRITICAL: 50.00% of data above the critical threshold [24.0]

1 0

** PROBLEM alert - labstore1005/High load average is CRITICAL **
by icinga＠einsteinium.wikimedia.org 26 Jul '18

26 Jul '18

Notification Type: PROBLEM Service: High load average Host: labstore1005 Address: 10.64.37.20 State: CRITICAL Date/Time: Thu Jul 26 17:14:11 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: CRITICAL: 50.00% of data above the critical threshold [24.0]

1 0

** RECOVERY alert - labstore1005/High load average is OK **
by icinga＠einsteinium.wikimedia.org 26 Jul '18

26 Jul '18

Notification Type: RECOVERY Service: High load average Host: labstore1005 Address: 10.64.37.20 State: OK Date/Time: Thu Jul 26 17:03:51 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: OK: Less than 50.00% above the threshold [16.0]

1 0

** PROBLEM alert - labstore1005/High load average is CRITICAL **
by icinga＠einsteinium.wikimedia.org 26 Jul '18

26 Jul '18

Notification Type: PROBLEM Service: High load average Host: labstore1005 Address: 10.64.37.20 State: CRITICAL Date/Time: Thu Jul 26 16:46:31 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: CRITICAL: 60.00% of data above the critical threshold [24.0]

1 0

** RECOVERY alert - labvirt1014/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 22 Jul '18

22 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labvirt1014 Address: 10.64.20.30 State: OK Date/Time: Sun Jul 22 06:55:54 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures

1 0

** PROBLEM alert - labvirt1014/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 22 Jul '18

22 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labvirt1014 Address: 10.64.20.30 State: CRITICAL Date/Time: Sun Jul 22 06:30:15 UTC 2018 Notes URLs: Additional Info: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/apt2xml]

1 0

** PROBLEM alert - labcontrol1003/keystone admin endpoint port 35357 is CRITICAL **
by icinga＠einsteinium.wikimedia.org 20 Jul '18

20 Jul '18

Notification Type: PROBLEM Service: keystone admin endpoint port 35357 Host: labcontrol1003 Address: 208.80.154.23 State: CRITICAL Date/Time: Fri Jul 20 09:20:53 UTC 2018 Notes URLs: Additional Info: connect to address 208.80.154.23 and port 35357: Connection refused

1 0

** PROBLEM alert - labcontrol1003/keystone public endoint port 5000 is CRITICAL **
by icinga＠einsteinium.wikimedia.org 20 Jul '18

20 Jul '18

Notification Type: PROBLEM Service: keystone public endoint port 5000 Host: labcontrol1003 Address: 208.80.154.23 State: CRITICAL Date/Time: Fri Jul 20 09:20:52 UTC 2018 Notes URLs: Additional Info: connect to address 208.80.154.23 and port 5000: Connection refused

1 0

** RECOVERY alert - labweb1002/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 19 Jul '18

19 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labweb1002 Address: 208.80.155.109 State: OK Date/Time: Thu Jul 19 13:33:07 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures

1 0

** RECOVERY alert - labweb1001/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 19 Jul '18

19 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labweb1001 Address: 208.80.154.160 State: OK Date/Time: Thu Jul 19 13:22:31 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures

1 0

** ACKNOWLEDGEMENT alert - labweb1002/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 19 Jul '18

19 Jul '18

Notification Type: ACKNOWLEDGEMENT Service: puppet last run Host: labweb1002 Address: 208.80.155.109 State: CRITICAL Date/Time: Thu Jul 19 07:05:33 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

** ACKNOWLEDGEMENT alert - labweb1001/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 19 Jul '18

19 Jul '18

Notification Type: ACKNOWLEDGEMENT Service: puppet last run Host: labweb1001 Address: 208.80.154.160 State: CRITICAL Date/Time: Thu Jul 19 07:05:31 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

** PROBLEM alert - labweb1002/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 19 Jul '18

19 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labweb1002 Address: 208.80.155.109 State: CRITICAL Date/Time: Thu Jul 19 07:04:31 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

** PROBLEM alert - labweb1001/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 19 Jul '18

19 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labweb1001 Address: 208.80.154.160 State: CRITICAL Date/Time: Thu Jul 19 06:54:00 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

** RECOVERY alert - labvirt1017/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 12 Jul '18

12 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labvirt1017 Address: 10.64.20.33 State: OK Date/Time: Thu Jul 12 07:00:26 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures

1 0

** PROBLEM alert - labvirt1017/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 12 Jul '18

12 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labvirt1017 Address: 10.64.20.33 State: CRITICAL Date/Time: Thu Jul 12 06:29:56 UTC 2018 Notes URLs: Additional Info: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/var/lib/apt/keys/ubuntucloud.gpg],File[/usr/local/share/ca-certificates/RapidSSL_SHA256_CA_-_G3.crt]

1 0

** RECOVERY alert - labvirt1017/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 10 Jul '18

10 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labvirt1017 Address: 10.64.20.33 State: OK Date/Time: Tue Jul 10 06:39:30 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures

1 0

** PROBLEM alert - labvirt1017/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 10 Jul '18

10 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labvirt1017 Address: 10.64.20.33 State: CRITICAL Date/Time: Tue Jul 10 06:29:30 UTC 2018 Notes URLs: Additional Info: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_sysctl],File[/usr/local/lib/nagios/plugins/check_long_procs]

1 0

** RECOVERY alert - labstore1007/High load average is OK **
by icinga＠einsteinium.wikimedia.org 08 Jul '18

08 Jul '18

Notification Type: RECOVERY Service: High load average Host: labstore1007 Address: 208.80.155.106 State: OK Date/Time: Sun Jul 8 19:42:09 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: OK: Less than 50.00% above the threshold [16.0]

1 0

** PROBLEM alert - labstore1007/High load average is CRITICAL **
by icinga＠einsteinium.wikimedia.org 08 Jul '18

08 Jul '18

Notification Type: PROBLEM Service: High load average Host: labstore1007 Address: 208.80.155.106 State: CRITICAL Date/Time: Sun Jul 8 18:53:38 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: CRITICAL: 77.78% of data above the critical threshold [24.0]

1 0

** RECOVERY alert - labcontrol1002/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 07 Jul '18

07 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labcontrol1002 Address: 208.80.154.95 State: OK Date/Time: Sat Jul 7 12:48:33 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures

1 0

** PROBLEM alert - labcontrol1002/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 07 Jul '18

07 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labcontrol1002 Address: 208.80.154.95 State: CRITICAL Date/Time: Sat Jul 7 12:43:32 UTC 2018 Notes URLs: Additional Info: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/gen_fingerprints]

1 0

** RECOVERY alert - labvirt1017/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 07 Jul '18

07 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labvirt1017 Address: 10.64.20.33 State: OK Date/Time: Sat Jul 7 06:36:53 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures

1 0

** PROBLEM alert - labvirt1017/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 07 Jul '18

07 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labvirt1017 Address: 10.64.20.33 State: CRITICAL Date/Time: Sat Jul 7 06:31:52 UTC 2018 Notes URLs: Additional Info: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/nova/policy.json],File[/usr/local/bin/apt2xml]

1 0

** RECOVERY alert - labstore1003/High load average is OK **
by icinga＠einsteinium.wikimedia.org 07 Jul '18

07 Jul '18

Notification Type: RECOVERY Service: High load average Host: labstore1003 Address: 10.64.4.10 State: OK Date/Time: Sat Jul 7 05:09:22 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: OK: Less than 50.00% above the threshold [16.0]

1 0

** PROBLEM alert - labstore1003/High load average is CRITICAL **
by icinga＠einsteinium.wikimedia.org 07 Jul '18

07 Jul '18

Notification Type: PROBLEM Service: High load average Host: labstore1003 Address: 10.64.4.10 State: CRITICAL Date/Time: Sat Jul 7 05:04:53 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/labs-monitoring Additional Info: CRITICAL: 50.00% of data above the critical threshold [24.0]

1 0

** RECOVERY alert - labnodepool1001/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 06 Jul '18

06 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labnodepool1001 Address: 10.64.20.18 State: OK Date/Time: Fri Jul 6 14:44:02 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures

1 0

** PROBLEM alert - labnodepool1001/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 06 Jul '18

06 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labnodepool1001 Address: 10.64.20.18 State: CRITICAL Date/Time: Fri Jul 6 14:28:43 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

** RECOVERY alert - labcontrol1002/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 04 Jul '18

04 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labcontrol1002 Address: 208.80.154.95 State: OK Date/Time: Wed Jul 4 11:29:26 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures

1 0

** RECOVERY alert - labcontrol1001/puppet last run is OK **
by icinga＠einsteinium.wikimedia.org 04 Jul '18

04 Jul '18

Notification Type: RECOVERY Service: puppet last run Host: labcontrol1001 Address: 208.80.154.92 State: OK Date/Time: Wed Jul 4 11:28:07 UTC 2018 Notes URLs: Additional Info: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures

1 0

** ACKNOWLEDGEMENT alert - labcontrol1002/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 04 Jul '18

04 Jul '18

Notification Type: ACKNOWLEDGEMENT Service: puppet last run Host: labcontrol1002 Address: 208.80.154.95 State: CRITICAL Date/Time: Wed Jul 4 11:18:22 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

** ACKNOWLEDGEMENT alert - labcontrol1001/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 04 Jul '18

04 Jul '18

Notification Type: ACKNOWLEDGEMENT Service: puppet last run Host: labcontrol1001 Address: 208.80.154.92 State: CRITICAL Date/Time: Wed Jul 4 11:15:17 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

** PROBLEM alert - labcontrol1002/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 04 Jul '18

04 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labcontrol1002 Address: 208.80.154.95 State: CRITICAL Date/Time: Wed Jul 4 11:14:12 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

** PROBLEM alert - labcontrol1001/puppet last run is CRITICAL **
by icinga＠einsteinium.wikimedia.org 04 Jul '18

04 Jul '18

Notification Type: PROBLEM Service: puppet last run Host: labcontrol1001 Address: 208.80.154.92 State: CRITICAL Date/Time: Wed Jul 4 11:07:52 UTC 2018 Notes URLs: Additional Info: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues

1 0

Host UP alert for labnodepool1002!
by icinga＠einsteinium.wikimedia.org 02 Jul '18

02 Jul '18

Notification Type: RECOVERY Host: labnodepool1002 State: UP Address: 10.64.20.37 Info: PING OK - Packet loss = 0%, RTA = 0.25 ms Date/Time: Mon Jul 2 15:34:31 UTC 2018

1 0

Host DOWN alert for labnodepool1002!
by icinga＠einsteinium.wikimedia.org 02 Jul '18

02 Jul '18

Notification Type: PROBLEM Host: labnodepool1002 State: DOWN Address: 10.64.20.37 Info: PING CRITICAL - Packet loss = 100% Date/Time: Mon Jul 2 15:33:21 UTC 2018

1 0

** RECOVERY alert - checker.tools.wmflabs.org/toolschecker: Start a job and verify on Trusty is OK **
by icinga＠einsteinium.wikimedia.org 01 Jul '18

01 Jul '18

Notification Type: RECOVERY Service: toolschecker: Start a job and verify on Trusty Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: OK Date/Time: Sun Jul 1 02:51:39 UTC 2018 Notes URLs: Additional Info: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 1.403 second response time

1 0

** PROBLEM alert - checker.tools.wmflabs.org/toolschecker: Start a job and verify on Trusty is CRITICAL **
by icinga＠einsteinium.wikimedia.org 01 Jul '18

01 Jul '18

Notification Type: PROBLEM Service: toolschecker: Start a job and verify on Trusty Host: checker.tools.wmflabs.org Address: checker.tools.wmflabs.org State: CRITICAL Date/Time: Sun Jul 1 02:47:29 UTC 2018 Notes URLs: Additional Info: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/trusty - 356 bytes in 60.007 second response time

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

Cloud-admin-feed July 2018