July 2017 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards is CRITICAL **
by icinga＠einsteinium.wikimedia.org 19 Jul '17

19 Jul '17

Notification Type: PROBLEM Service: ElasticSearch health check for shards Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Jul 19 08:23:12 UTC 2017 Additional Info: CRITICAL - elasticsearch http://10.2.1.30:9200/_cluster/health error while fetching: HTTPConnectionPool(host=10.2.1.30, port=9200): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 13 Jul '17

13 Jul '17

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Fri Jul 14 00:01:22 UTC 2017 Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 13 Jul '17

13 Jul '17

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Thu Jul 13 23:29:12 UTC 2017 Additional Info: CRITICAL: 30.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 13 Jul '17

13 Jul '17

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Thu Jul 13 23:24:12 UTC 2017 Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 13 Jul '17

13 Jul '17

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Thu Jul 13 23:02:57 UTC 2017 Additional Info: CRITICAL: 50.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards is OK **
by icinga＠einsteinium.wikimedia.org 12 Jul '17

12 Jul '17

Notification Type: RECOVERY Service: ElasticSearch health check for shards Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Wed Jul 12 08:31:09 UTC 2017 Additional Info: OK - elasticsearch status production-search-codfw: status: yellow, number_of_nodes: 36, unassigned_shards: 770, number_of_pending_tasks: 1680, number_of_in_flight_fetch: 0, timed_out: False, active_primary_shards: 3081, task_max_waiting_in_queue_millis: 171073, cluster_name: production-search-codfw, relocating_shards: 0, active_shards_percent_as_number: 91.6621548457, active_shards: 8465, initializing_shards: 0, number_of_data_nodes: 36, delayed_unassigned_shards: 770

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards is CRITICAL **
by icinga＠einsteinium.wikimedia.org 12 Jul '17

12 Jul '17

Notification Type: PROBLEM Service: ElasticSearch health check for shards Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Jul 12 08:24:19 UTC 2017 Additional Info: CRITICAL - elasticsearch http://10.2.1.30:9200/_cluster/health error while fetching: HTTPConnectionPool(host=10.2.1.30, port=9200): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 11 Jul '17

11 Jul '17

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Tue Jul 11 18:35:57 UTC 2017 Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** RECOVERY alert - graphite1001/Number of backend failures per minute from CirrusSearch is OK **
by icinga＠einsteinium.wikimedia.org 11 Jul '17

11 Jul '17

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Tue Jul 11 18:34:27 UTC 2017 Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1001/Number of backend failures per minute from CirrusSearch is CRITICAL **
by icinga＠einsteinium.wikimedia.org 11 Jul '17

11 Jul '17

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Tue Jul 11 18:29:17 UTC 2017 Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts July 2017