May 2022 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - relforge1003/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 May '22

23 May '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Mon May 23 23:56:22 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 96 threshold =0.15 breach: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 170, active_shards: 211, relocating_shards: 0, initializing_shards: 2, unassigned_shards: 94, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 68.72964169381108

1 0

** RECOVERY alert - relforge1004/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 23 May '22

23 May '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: OK Date/Time: Mon May 23 22:07:55 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad-small-alpha: relocating_shards: 0, status: green, number_of_nodes: 2, cluster_name: relforge-eqiad-small-alpha, delayed_unassigned_shards: 0, initializing_shards: 0, timed_out: False, active_shards_percent_as_number: 100.0, unassigned_shards: 0, active_primary_shards: 37, task_max_waiting_in_queue_millis: 0, number_of_pending_tasks: 0, active_shards: 42, number_of_in_flight_fetch: 0, number_of_data_nodes: 2

1 0

** ACKNOWLEDGEMENT alert - relforge1004/ElasticSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 May '22

23 May '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Mon May 23 21:48:03 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Ryan Kemper: T308770 Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** ACKNOWLEDGEMENT alert - relforge1004/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 May '22

23 May '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Mon May 23 21:48:03 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Ryan Kemper: T308770 Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 May '22

23 May '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Mon May 23 21:47:57 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 May '22

23 May '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Mon May 23 21:46:23 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 23 May '22

23 May '22

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Mon May 23 06:48:01 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.16.50 elastic1076:9108 elasticsearch elastic1076-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 23 May '22

23 May '22

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 May '22

23 May '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Mon May 23 06:13:49 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [1000.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.32.29 elastic1080:9108 elasticsearch elastic1080-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 22 May '22

22 May '22

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts May 2022