August 2018 - Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Tue Aug 14 13:29:20 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Tue Aug 14 13:13:50 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: CRITICAL: 50.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - graphite1001/Number of backend failures per minute from CirrusSearch is OK **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Tue Aug 14 12:42:59 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1001/Number of backend failures per minute from CirrusSearch is CRITICAL **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Tue Aug 14 12:36:48 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards is OK **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: RECOVERY Service: ElasticSearch health check for shards Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Tue Aug 14 12:32:28 UTC 2018 Notes URLs: Additional Info: OK - elasticsearch status production-search-eqiad: status: yellow, number_of_nodes: 32, unassigned_shards: 141, number_of_pending_tasks: 589, number_of_in_flight_fetch: 0, timed_out: False, active_primary_shards: 3139, task_max_waiting_in_queue_millis: 65118, cluster_name: production-search-eqiad, relocating_shards: 61, active_shards_percent_as_number: 98.3982178848, active_shards: 9276, initializing_shards: 10, number_of_data_nodes: 32, delayed_unassigned_shards: 141

1 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards is CRITICAL **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: PROBLEM Service: ElasticSearch health check for shards Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Tue Aug 14 12:31:28 UTC 2018 Notes URLs: Additional Info: CRITICAL - elasticsearch http://10.2.2.30:9200/_cluster/health error while fetching: HTTPConnectionPool(host=10.2.2.30, port=9200): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Tue Aug 14 12:03:59 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Tue Aug 14 11:40:59 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Tue Aug 14 09:58:04 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 14 Aug '18

14 Aug '18

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Tue Aug 14 09:33:23 UTC 2018 Notes URLs: https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts August 2018