June 2024 - Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - logstash1036/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1036 Address: 10.64.130.12 State: OK Date/Time: Sat Jun 22 22:29:16 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1227, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 492, delayed_unassigned_shards: 0, number_of_pending_tasks: 8, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1243, active_shards_percent_as_number: 70.96587622903412

1 0

** RECOVERY alert - logstash1034/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1034 Address: 10.64.32.112 State: OK Date/Time: Sat Jun 22 22:29:15 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1223, relocating_shards: 0, initializing_shards: 14, unassigned_shards: 492, delayed_unassigned_shards: 0, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 888, active_shards_percent_as_number: 70.73452862926547

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9243 is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Sat Jun 22 22:29:13 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, active_primary_shards: 1381, active_shards: 3490, relocating_shards: 0, initializing_shards: 103, unassigned_shards: 598, delayed_unassigned_shards: 0, number_of_pending_tasks: 103, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 293284, active_shards_percent_as_number: 83.27368169887855

1 0

** RECOVERY alert - logstash1033/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1033 Address: 10.64.0.87 State: OK Date/Time: Sat Jun 22 22:29:07 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1200, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 519, delayed_unassigned_shards: 0, number_of_pending_tasks: 8, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 2316, active_shards_percent_as_number: 69.40427993059572

1 0

** RECOVERY alert - cloudelastic1007/ElasticSearch health check for shards on 9600 is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9600 Host: cloudelastic1007 Address: 10.64.0.96 State: OK Date/Time: Sat Jun 22 22:29:07 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, active_primary_shards: 791, active_shards: 1368, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 213, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 86.52751423149905

1 0

** RECOVERY alert - logstash1029/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1029 Address: 10.64.48.120 State: OK Date/Time: Sat Jun 22 22:29:05 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1196, relocating_shards: 0, initializing_shards: 14, unassigned_shards: 519, delayed_unassigned_shards: 0, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 765, active_shards_percent_as_number: 69.17293233082707

1 0

** RECOVERY alert - logstash1028/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1028 Address: 10.64.32.104 State: OK Date/Time: Sat Jun 22 22:29:05 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1196, relocating_shards: 0, initializing_shards: 14, unassigned_shards: 519, delayed_unassigned_shards: 0, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 732, active_shards_percent_as_number: 69.17293233082707

1 0

** PROBLEM alert - an-airflow1005/Checks that the local airflow scheduler for airflow @search is working properly is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: PROBLEM Service: Checks that the local airflow scheduler for airflow @search is working properly Host: an-airflow1005 Address: 10.64.36.11 State: CRITICAL Date/Time: Sat Jun 22 22:28:41 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow Acknowledged by : Additional Info: CRITICAL: /usr/bin/env PYTHONPATH=/srv/deployment/airflow-dags/search AIRFLOW_HOME=/srv/airflow-search /usr/lib/airflow/bin/airflow jobs check --job-type SchedulerJob --hostname an-airflow1005.eqiad.wmnet did not succeed

1 0

** PROBLEM alert - logging-hd1002/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: logging-hd1002 Address: 10.64.16.76 State: CRITICAL Date/Time: Sat Jun 22 22:28:33 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 635 threshold =0.34 breach: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 17, number_of_data_nodes: 11, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1093, relocating_shards: 0, initializing_shards: 12, unassigned_shards: 623, delayed_unassigned_shards: 0, number_of_pending_tasks: 8, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 2534, active_shards_percent_as_number: 63.25231481481482

1 0

** PROBLEM alert - logstash1027/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: logstash1027 Address: 10.64.16.169 State: CRITICAL Date/Time: Sat Jun 22 22:28:31 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 631 threshold =0.34 breach: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 17, number_of_data_nodes: 11, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1097, relocating_shards: 0, initializing_shards: 8, unassigned_shards: 623, delayed_unassigned_shards: 0, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 3017, active_shards_percent_as_number: 63.48379629629629

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts June 2024