April 2020 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - cloudelastic1004/Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: PROBLEM Service: Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad Host: cloudelastic1004 Address: 208.80.155.107 State: CRITICAL Date/Time: Fri Apr 17 20:02:26 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_… https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-… Acknowledged by : Additional Info: 100.9 gt 100

1 0

** RECOVERY alert - cloudelastic1004/Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad is OK **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: RECOVERY Service: Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad Host: cloudelastic1004 Address: 208.80.155.107 State: OK Date/Time: Fri Apr 17 19:42:04 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_… https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-… Acknowledged by : Additional Info: (C)100 gt (W)80 gt 14.84

1 0

** RECOVERY alert - cloudelastic1001/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: OK Date/Time: Fri Apr 17 19:37:06 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: active_shards_percent_as_number: 98.2769623484365, cluster_name: cloudelastic-chi-eqiad, number_of_data_nodes: 4, number_of_nodes: 4, unassigned_shards: 22, delayed_unassigned_shards: 0, active_shards: 1540, status: yellow, timed_out: False, initializing_shards: 5, task_max_waiting_in_queue_millis: 0, number_of_in_flight_fetch: 0, active_primary_shards: 776, relocating_shards: 0, number_of_pending_tasks: 0

1 0

** RECOVERY alert - cloudelastic1003/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1003 Address: 208.80.154.83 State: OK Date/Time: Fri Apr 17 19:36:42 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: timed_out: False, initializing_shards: 5, number_of_pending_tasks: 2, active_shards: 1527, task_max_waiting_in_queue_millis: 141, unassigned_shards: 35, number_of_data_nodes: 4, delayed_unassigned_shards: 0, relocating_shards: 0, status: yellow, cluster_name: cloudelastic-chi-eqiad, active_shards_percent_as_number: 97.44735162731334, active_primary_shards: 776, number_of_nodes: 4, number_of_in_flight_fetch: 0

1 0

** RECOVERY alert - cloudelastic1004/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1004 Address: 208.80.155.107 State: OK Date/Time: Fri Apr 17 19:36:20 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: unassigned_shards: 46, active_shards: 1516, number_of_nodes: 4, status: yellow, number_of_data_nodes: 4, timed_out: False, relocating_shards: 0, active_shards_percent_as_number: 96.7453733248245, active_primary_shards: 776, number_of_in_flight_fetch: 0, delayed_unassigned_shards: 0, cluster_name: cloudelastic-chi-eqiad, task_max_waiting_in_queue_millis: 0, number_of_pending_tasks: 0, initializing_shards: 5

1 0

** RECOVERY alert - cloudelastic1002/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: OK Date/Time: Fri Apr 17 19:35:48 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: delayed_unassigned_shards: 0, number_of_nodes: 4, active_shards: 1449, timed_out: False, task_max_waiting_in_queue_millis: 29, initializing_shards: 5, number_of_data_nodes: 4, cluster_name: cloudelastic-chi-eqiad, relocating_shards: 0, active_shards_percent_as_number: 92.46968730057435, number_of_pending_tasks: 2, active_primary_shards: 776, status: yellow, number_of_in_flight_fetch: 0, unassigned_shards: 113

1 0

** PROBLEM alert - cloudelastic1001/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: CRITICAL Date/Time: Fri Apr 17 19:33:28 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 370 threshold =0.15 breach: number_of_data_nodes: 3, task_max_waiting_in_queue_millis: 757, number_of_nodes: 3, status: yellow, number_of_in_flight_fetch: 0, relocating_shards: 0, unassigned_shards: 358, active_shards: 1197, timed_out: False, active_primary_shards: 776, cluster_name: cloudelastic-chi-eqiad, active_shards_percent_as_number: 76.38800255264837, number_of_pending_tasks: 2, delayed_unassigned_shards: 0, initializing_shards: 12

1 0

** PROBLEM alert - cloudelastic1003/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1003 Address: 208.80.154.83 State: CRITICAL Date/Time: Fri Apr 17 19:33:10 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 374 threshold =0.15 breach: number_of_nodes: 3, relocating_shards: 0, number_of_data_nodes: 3, active_primary_shards: 776, status: yellow, delayed_unassigned_shards: 0, unassigned_shards: 362, initializing_shards: 12, timed_out: False, cluster_name: cloudelastic-chi-eqiad, active_shards_percent_as_number: 76.1327377153797, active_shards: 1193, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1537, number_of_pending_tasks: 2

1 0

** PROBLEM alert - cloudelastic1002/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: CRITICAL Date/Time: Fri Apr 17 19:32:10 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 381 threshold =0.15 breach: active_shards: 1186, active_primary_shards: 776, task_max_waiting_in_queue_millis: 0, unassigned_shards: 369, number_of_data_nodes: 3, number_of_nodes: 3, status: yellow, delayed_unassigned_shards: 0, initializing_shards: 12, relocating_shards: 0, active_shards_percent_as_number: 75.68602425015955, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, timed_out: False, cluster_name: cloudelastic-chi-eqiad

1 0

** PROBLEM alert - cloudelastic1004/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Apr '20

17 Apr '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1004 Address: 208.80.155.107 State: CRITICAL Date/Time: Fri Apr 17 19:30:52 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Max retries exceeded with url: /_cluster/health (Caused by NewConnectionError(requests.packages.urllib3.connection.HTTPConnection object at 0x7f4b4afba470: Failed to establish a new connection: [Errno 111] Connection refused,))

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts April 2020