October 2023 - Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - graphite1005/CirrusSearch more_like codfw 95th percentile latency is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: CirrusSearch more_like codfw 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Tue Oct 24 06:08:23 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [1000.0]

7 months, 1 week

1
0
0 0

** PROBLEM alert - graphite1005/CirrusSearch more_like codfw 95th percentile latency is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: CirrusSearch more_like codfw 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Tue Oct 24 06:02:11 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [1500.0]

7 months, 1 week

1
0
0 0

** RECOVERY alert - logstash1023/OpenSearch health check for shards on 9200 is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1023 Address: 10.64.0.183 State: OK Date/Time: Tue Oct 24 00:34:41 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: green, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 646, active_shards: 1494, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

7 months, 1 week

1
0
0 0

** PROBLEM alert - logstash1023/OpenSearch health check for shards on 9200 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: logstash1023 Address: 10.64.0.183 State: CRITICAL Date/Time: Tue Oct 24 00:33:19 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Max retries exceeded with url: /_cluster/health (Caused by NewConnectionError(urllib3.connection.HTTPConnection object at 0x7f926e15b280: Failed to establish a new connection: [Errno 111] Connection refused))

7 months, 1 week