- Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards is OK **
by icinga＠einsteinium.wikimedia.org 29 Jun '17

29 Jun '17

Notification Type: RECOVERY Service: ElasticSearch health check for shards Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Thu Jun 29 09:31:54 UTC 2017 Additional Info: OK - elasticsearch status production-search-codfw: status: red, number_of_nodes: 27, unassigned_shards: 843, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, timed_out: False, active_primary_shards: 3080, task_max_waiting_in_queue_millis: 934, cluster_name: production-search-codfw, relocating_shards: 0, active_shards_percent_as_number: 90.1266645015, active_shards: 8325, initializing_shards: 69, number_of_data_nodes: 27, delayed_unassigned_shards: 0

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards is CRITICAL **
by icinga＠einsteinium.wikimedia.org 29 Jun '17

29 Jun '17

Notification Type: PROBLEM Service: ElasticSearch health check for shards Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Thu Jun 29 09:04:04 UTC 2017 Additional Info: CRITICAL - elasticsearch inactive shards 2198 threshold =0.1 breach: status: red, number_of_nodes: 27, unassigned_shards: 2094, number_of_pending_tasks: 2364, number_of_in_flight_fetch: 0, timed_out: False, active_primary_shards: 3077, task_max_waiting_in_queue_millis: 674059, cluster_name: production-search-codfw, relocating_shards: 0, active_shards_percent_as_number: 76.1966645008, active_shards: 7036, initializing_shards: 104, number_of_data_nodes: 27, delayed_unassigned_shards: 0

1 0

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 27 Jun '17

27 Jun '17

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Tue Jun 27 14:59:54 UTC 2017 Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 27 Jun '17

27 Jun '17

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Tue Jun 27 14:50:54 UTC 2017 Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 26 Jun '17

26 Jun '17

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Mon Jun 26 20:29:25 UTC 2017 Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 26 Jun '17

26 Jun '17

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Mon Jun 26 19:14:25 UTC 2017 Additional Info: CRITICAL: 30.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - graphite1001/CirrusSearch eqiad 95th percentile latency is OK **
by icinga＠einsteinium.wikimedia.org 26 Jun '17

26 Jun '17

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: OK Date/Time: Mon Jun 26 16:42:25 UTC 2017 Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1001/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by icinga＠einsteinium.wikimedia.org 26 Jun '17

26 Jun '17

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1001 Address: 10.64.32.155 State: CRITICAL Date/Time: Mon Jun 26 16:38:25 UTC 2017 Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards is OK **
by icinga＠einsteinium.wikimedia.org 20 Jun '17

20 Jun '17

Notification Type: RECOVERY Service: ElasticSearch health check for shards Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Jun 20 16:24:44 UTC 2017 Additional Info: OK - elasticsearch status production-search-codfw: status: yellow, number_of_nodes: 36, unassigned_shards: 847, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, timed_out: False, active_primary_shards: 3072, task_max_waiting_in_queue_millis: 0, cluster_name: production-search-codfw, relocating_shards: 0, active_shards_percent_as_number: 90.2809415338, active_shards: 8323, initializing_shards: 49, number_of_data_nodes: 36, delayed_unassigned_shards: 0

1 0

** ACKNOWLEDGEMENT alert - search.svc.codfw.wmnet/ElasticSearch health check for shards is CRITICAL **
by icinga＠einsteinium.wikimedia.org 20 Jun '17

20 Jun '17

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Jun 20 16:22:00 UTC 2017 Additional Info: CRITICAL - elasticsearch inactive shards 1096 threshold =0.1 breach: status: yellow, number_of_nodes: 31, unassigned_shards: 1010, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, timed_out: False, active_primary_shards: 3072, task_max_waiting_in_queue_millis: 947, cluster_name: production-search-codfw, relocating_shards: 0, active_shards_percent_as_number: 88.1115088404, active_shards: 8123, initializing_shards: 86, number_of_data_nodes: 31, delayed_unassigned_shards: 0

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts