August 2018 - Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 13 Aug '18

13 Aug '18

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Mon Aug 13 18:29:53 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** ACKNOWLEDGEMENT alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 13 Aug '18

13 Aug '18

Notification Type: ACKNOWLEDGEMENT Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Mon Aug 13 17:20:02 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: CRITICAL: 100.00% of data above the critical threshold [1000.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 13 Aug '18

13 Aug '18

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Mon Aug 13 16:52:43 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - graphite1001/Number of backend failures per minute from CirrusSearch is OK **
by icinga＠einsteinium.wikimedia.org 13 Aug '18

13 Aug '18

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Mon Aug 13 13:42:29 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1001/Number of backend failures per minute from CirrusSearch is CRITICAL **
by icinga＠einsteinium.wikimedia.org 13 Aug '18

13 Aug '18

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Mon Aug 13 13:36:28 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

Fatal Error - Oozie Job discovery-transfer_to_es-discovery.popularity_score-2018, 8, 6->cirrussearch-wf
by oozie＠analytics1003.eqiad.wmnet 10 Aug '18

10 Aug '18