September 2019 - Discovery-alerts

** RECOVERY alert - cloudelastic1001/ElasticSearch health check for shards on 9200 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: OK Date/Time: Sat Sept 21 00:27:33 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: number_of_nodes: 3, cluster_name: cloudelastic-chi-eqiad, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.67530695770806, number_of_in_flight_fetch: 0, unassigned_shards: 198, timed_out: False, initializing_shards: 12, relocating_shards: 0, status: yellow, delayed_unassigned_shards: 0, number_of_data_nodes: 3, active_shards: 1256, active_primary_shards: 733, number_of_pending_tasks: 0

3 years, 6 months

1
0
0 0

** RECOVERY alert - cloudelastic1002/ElasticSearch health check for shards on 9200 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: OK Date/Time: Sat Sept 21 00:26:33 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: active_primary_shards: 733, timed_out: False, number_of_data_nodes: 3, task_max_waiting_in_queue_millis: 0, number_of_in_flight_fetch: 0, cluster_name: cloudelastic-chi-eqiad, status: yellow, number_of_nodes: 3, active_shards_percent_as_number: 85.19781718963165, active_shards: 1249, number_of_pending_tasks: 0, unassigned_shards: 205, relocating_shards: 0, delayed_unassigned_shards: 0, initializing_shards: 12

3 years, 6 months

1
0
0 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Sat Sept 21 00:09:27 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [300.0]

3 years, 6 months

1
0
0 0

** RECOVERY alert - cloudelastic1001/ElasticSearch health check for shards on 9400 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1001 Address: 208.80.154.8 State: OK Date/Time: Sat Sept 21 00:06:41 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-omega-eqiad: number_of_data_nodes: 3, unassigned_shards: 263, initializing_shards: 4, active_shards_percent_as_number: 87.15728715728716, number_of_pending_tasks: 0, status: yellow, number_of_in_flight_fetch: 0, active_primary_shards: 693, active_shards: 1812, relocating_shards: 0, number_of_nodes: 3, timed_out: False, cluster_name: cloudelastic-omega-eqiad, task_max_waiting_in_queue_millis: 0, delayed_unassigned_shards: 0

3 years, 6 months

1
0
0 0

** RECOVERY alert - cloudelastic1002/ElasticSearch health check for shards on 9400 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1002 Address: 208.80.154.133 State: OK Date/Time: Sat Sept 21 00:05:37 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-omega-eqiad: active_shards_percent_as_number: 85.37758537758538, number_of_nodes: 3, task_max_waiting_in_queue_millis: 0, timed_out: False, number_of_in_flight_fetch: 0, number_of_data_nodes: 3, active_shards: 1775, status: yellow, cluster_name: cloudelastic-omega-eqiad, initializing_shards: 4, active_primary_shards: 693, delayed_unassigned_shards: 0, unassigned_shards: 300, relocating_shards: 0, number_of_pending_tasks: 0

3 years, 6 months

1
0
0 0

** RECOVERY alert - cloudelastic1003/ElasticSearch health check for shards on 9400 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1003 Address: 208.80.154.83 State: OK Date/Time: Sat Sept 21 00:05:35 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-omega-eqiad: number_of_nodes: 3, number_of_data_nodes: 3, active_primary_shards: 693, status: yellow, number_of_pending_tasks: 1, unassigned_shards: 302, active_shards_percent_as_number: 85.28138528138528, timed_out: False, relocating_shards: 0, active_shards: 1773, task_max_waiting_in_queue_millis: 0, initializing_shards: 4, delayed_unassigned_shards: 0, number_of_in_flight_fetch: 0, cluster_name: cloudelastic-omega-eqiad

3 years, 6 months

1
0
0 0

** RECOVERY alert - cloudelastic1001/ElasticSearch health check for shards on 9600 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9600 Host: cloudelastic1001 Address: 208.80.154.8 State: OK Date/Time: Sat Sept 21 00:04:41 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-psi-eqiad: unassigned_shards: 295, cluster_name: cloudelastic-psi-eqiad, initializing_shards: 4, active_shards: 1819, active_shards_percent_as_number: 85.88290840415486, delayed_unassigned_shards: 0, number_of_in_flight_fetch: 0, timed_out: False, task_max_waiting_in_queue_millis: 0, status: yellow, number_of_data_nodes: 3, relocating_shards: 0, active_primary_shards: 706, number_of_nodes: 3, number_of_pending_tasks: 0

3 years, 6 months

1
0
0 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sat Sept 21 00:04:35 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

3 years, 6 months

1
0
0 0

** RECOVERY alert - cloudelastic1002/ElasticSearch health check for shards on 9600 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9600 Host: cloudelastic1002 Address: 208.80.154.133 State: OK Date/Time: Sat Sept 21 00:04:21 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-psi-eqiad: status: yellow, task_max_waiting_in_queue_millis: 0, number_of_in_flight_fetch: 0, unassigned_shards: 305, number_of_nodes: 3, active_primary_shards: 706, number_of_data_nodes: 3, delayed_unassigned_shards: 0, timed_out: False, initializing_shards: 4, relocating_shards: 0, number_of_pending_tasks: 1, cluster_name: cloudelastic-psi-eqiad, active_shards: 1809, active_shards_percent_as_number: 85.41076487252126

3 years, 6 months

1
0
0 0

** PROBLEM alert - cloudelastic1002/ElasticSearch health check for shards on 9200 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: CRITICAL Date/Time: Sat Sept 21 00:04:05 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 357 threshold =0.15 breach: active_shards_percent_as_number: 75.64802182810368, timed_out: False, number_of_pending_tasks: 3, active_primary_shards: 733, delayed_unassigned_shards: 0, number_of_in_flight_fetch: 0, cluster_name: cloudelastic-chi-eqiad, initializing_shards: 12, unassigned_shards: 345, number_of_nodes: 3, number_of_data_nodes: 3, task_max_waiting_in_queue_millis: 7711, active_shards: 1109, relocating_shards: 0, status: yellow

3 years, 6 months

1
0
0 0

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts September 2019