April 2020 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - cloudelastic1003/Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad is CRITICAL **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: PROBLEM Service: Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad Host: cloudelastic1003 Address: 208.80.154.83 State: CRITICAL Date/Time: Sat Apr 18 16:51:52 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_… https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-… Acknowledged by : Additional Info: 109.1 gt 100

1 0

** RECOVERY alert - cloudelastic1003/Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad is OK **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: RECOVERY Service: Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad Host: cloudelastic1003 Address: 208.80.154.83 State: OK Date/Time: Sat Apr 18 16:02:08 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_… https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-… Acknowledged by : Additional Info: (C)100 gt (W)80 gt 1.404

1 0

** RECOVERY alert - cloudelastic1004/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1004 Address: 208.80.155.107 State: OK Date/Time: Sat Apr 18 16:01:08 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: initializing_shards: 5, number_of_data_nodes: 4, relocating_shards: 0, timed_out: False, active_shards_percent_as_number: 96.36475662353666, number_of_pending_tasks: 4, active_primary_shards: 804, active_shards: 1564, number_of_nodes: 4, unassigned_shards: 54, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 10195, cluster_name: cloudelastic-chi-eqiad, delayed_unassigned_shards: 0, status: yellow

1 0

** RECOVERY alert - cloudelastic1002/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: OK Date/Time: Sat Apr 18 16:01:04 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: active_primary_shards: 804, relocating_shards: 0, cluster_name: cloudelastic-chi-eqiad, number_of_in_flight_fetch: 0, unassigned_shards: 54, number_of_pending_tasks: 4, active_shards: 1564, number_of_data_nodes: 4, task_max_waiting_in_queue_millis: 4312, delayed_unassigned_shards: 0, number_of_nodes: 4, active_shards_percent_as_number: 96.36475662353666, status: yellow, timed_out: False, initializing_shards: 5

1 0

** RECOVERY alert - cloudelastic1001/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: OK Date/Time: Sat Apr 18 16:00:36 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: timed_out: False, number_of_nodes: 4, delayed_unassigned_shards: 0, active_primary_shards: 804, unassigned_shards: 96, initializing_shards: 5, relocating_shards: 0, number_of_data_nodes: 4, number_of_pending_tasks: 4, active_shards_percent_as_number: 93.7769562538509, cluster_name: cloudelastic-chi-eqiad, number_of_in_flight_fetch: 0, active_shards: 1522, task_max_waiting_in_queue_millis: 617, status: yellow

1 0

** RECOVERY alert - cloudelastic1003/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1003 Address: 208.80.154.83 State: OK Date/Time: Sat Apr 18 16:00:08 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: number_of_pending_tasks: 0, number_of_nodes: 4, cluster_name: cloudelastic-chi-eqiad, number_of_in_flight_fetch: 0, active_shards_percent_as_number: 89.40234134319162, unassigned_shards: 166, number_of_data_nodes: 4, relocating_shards: 0, active_primary_shards: 804, initializing_shards: 6, delayed_unassigned_shards: 0, status: yellow, task_max_waiting_in_queue_millis: 0, active_shards: 1451, timed_out: False

1 0

** PROBLEM alert - cloudelastic1001/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: CRITICAL Date/Time: Sat Apr 18 15:47:38 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 388 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, timed_out: False, active_primary_shards: 797, initializing_shards: 12, relocating_shards: 0, number_of_data_nodes: 3, status: yellow, active_shards_percent_as_number: 75.88564325668116, unassigned_shards: 376, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, delayed_unassigned_shards: 0, active_shards: 1221, task_max_waiting_in_queue_millis: 0, number_of_nodes: 3

1 0

** PROBLEM alert - cloudelastic1004/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1004 Address: 208.80.155.107 State: CRITICAL Date/Time: Sat Apr 18 15:46:22 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 390 threshold =0.15 breach: status: yellow, timed_out: False, active_shards_percent_as_number: 75.76134244872593, initializing_shards: 12, delayed_unassigned_shards: 0, active_primary_shards: 797, number_of_nodes: 3, task_max_waiting_in_queue_millis: 0, number_of_in_flight_fetch: 0, unassigned_shards: 378, active_shards: 1219, number_of_data_nodes: 3, relocating_shards: 0, number_of_pending_tasks: 0, cluster_name: cloudelastic-chi-eqiad

1 0

** PROBLEM alert - cloudelastic1002/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: CRITICAL Date/Time: Sat Apr 18 15:46:16 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 390 threshold =0.15 breach: status: yellow, number_of_nodes: 3, delayed_unassigned_shards: 0, active_primary_shards: 797, cluster_name: cloudelastic-chi-eqiad, unassigned_shards: 378, number_of_data_nodes: 3, task_max_waiting_in_queue_millis: 0, relocating_shards: 0, timed_out: False, number_of_in_flight_fetch: 0, active_shards: 1219, initializing_shards: 12, active_shards_percent_as_number: 75.76134244872593, number_of_pending_tasks: 0

1 0

** PROBLEM alert - cloudelastic1003/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 18 Apr '20

18 Apr '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1003 Address: 208.80.154.83 State: CRITICAL Date/Time: Sat Apr 18 15:45:26 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Max retries exceeded with url: /_cluster/health (Caused by NewConnectionError(requests.packages.urllib3.connection.HTTPConnection object at 0x7fe4657c8400: Failed to establish a new connection: [Errno 111] Connection refused,))

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts April 2020