August 2026 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - cirrussearch1108/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1108 Address: 10.64.130.24 State: CRITICAL Date/Time: Wed Aug 5 16:21:39 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1095/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1095 Address: 10.64.132.5 State: CRITICAL Date/Time: Wed Aug 5 16:21:39 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1075/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1075 Address: 10.64.16.49 State: CRITICAL Date/Time: Wed Aug 5 16:21:39 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1122/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1122 Address: 10.64.187.9 State: CRITICAL Date/Time: Wed Aug 5 16:21:39 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1102/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1102 Address: 10.64.136.6 State: CRITICAL Date/Time: Wed Aug 5 16:21:39 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1099/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1099 Address: 10.64.135.6 State: CRITICAL Date/Time: Wed Aug 5 16:21:39 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1069/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1069 Address: 10.64.0.73 State: CRITICAL Date/Time: Wed Aug 5 16:21:37 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1086/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1086 Address: 10.64.16.64 State: CRITICAL Date/Time: Wed Aug 5 16:21:37 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1078/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1078 Address: 10.64.16.52 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1073/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1073 Address: 10.64.0.78 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1114/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1114 Address: 10.64.16.133 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1072/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1072 Address: 10.64.0.77 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1124/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1124 Address: 10.64.166.2 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1125/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1125 Address: 10.64.166.3 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1112/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1112 Address: 10.64.0.175 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1120/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1120 Address: 10.64.48.29 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1113/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1113 Address: 10.64.16.37 State: CRITICAL Date/Time: Wed Aug 5 16:21:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1077/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1077 Address: 10.64.16.51 State: CRITICAL Date/Time: Wed Aug 5 16:21:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1103/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1103 Address: 10.64.183.7 State: CRITICAL Date/Time: Wed Aug 5 16:21:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1109/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1109 Address: 10.64.152.2 State: CRITICAL Date/Time: Wed Aug 5 16:21:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1107/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1107 Address: 10.64.160.2 State: CRITICAL Date/Time: Wed Aug 5 16:21:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1091/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1091 Address: 10.64.131.4 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1070/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1070 Address: 10.64.0.74 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1098/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1098 Address: 10.64.135.5 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1094/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1094 Address: 10.64.132.4 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1089/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1089 Address: 10.64.130.7 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1096/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1096 Address: 10.64.134.6 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1100/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1100 Address: 10.64.136.4 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1080/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1080 Address: 10.64.169.8 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1093/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1093 Address: 10.64.132.2 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1068/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1068 Address: 10.64.0.72 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1071/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1071 Address: 10.64.0.76 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1082/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1082 Address: 10.64.175.4 State: CRITICAL Date/Time: Wed Aug 5 16:21:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1074/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1074 Address: 10.64.16.42 State: CRITICAL Date/Time: Wed Aug 5 16:21:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1076/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch1076 Address: 10.64.16.50 State: CRITICAL Date/Time: Wed Aug 5 16:21:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9243 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Wed Aug 5 16:16:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3866, relocating_shards: 0, initializing_shards: 8, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.79349509550852

1 0

** RECOVERY alert - cirrussearch1117/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1117 Address: 10.64.16.139 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1121/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1121 Address: 10.64.187.5 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1113/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1113 Address: 10.64.16.37 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1120/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1120 Address: 10.64.48.29 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1123/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1123 Address: 10.64.158.2 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1112/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1112 Address: 10.64.0.175 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1111/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1111 Address: 10.64.0.152 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1115/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1115 Address: 10.64.16.136 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1116/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1116 Address: 10.64.16.138 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1114/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1114 Address: 10.64.16.133 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1124/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1124 Address: 10.64.166.2 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1101/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1101 Address: 10.64.136.5 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1097/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1097 Address: 10.64.134.7 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1109/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1109 Address: 10.64.152.2 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1107/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1107 Address: 10.64.160.2 State: OK Date/Time: Wed Aug 5 16:16:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3864, relocating_shards: 0, initializing_shards: 10, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.74186886938566

1 0

** RECOVERY alert - cirrussearch1091/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1091 Address: 10.64.131.4 State: OK Date/Time: Wed Aug 5 16:16:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 2191, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1125/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1125 Address: 10.64.166.3 State: OK Date/Time: Wed Aug 5 16:16:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 2093, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1103/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1103 Address: 10.64.183.7 State: OK Date/Time: Wed Aug 5 16:16:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 2053, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1083/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1083 Address: 10.64.175.7 State: OK Date/Time: Wed Aug 5 16:16:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1096/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1096 Address: 10.64.134.6 State: OK Date/Time: Wed Aug 5 16:16:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1110/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1110 Address: 10.64.134.22 State: OK Date/Time: Wed Aug 5 16:16:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1099/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1099 Address: 10.64.135.6 State: OK Date/Time: Wed Aug 5 16:16:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1092/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1092 Address: 10.64.131.6 State: OK Date/Time: Wed Aug 5 16:16:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1108/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1108 Address: 10.64.130.24 State: OK Date/Time: Wed Aug 5 16:16:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1084/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1084 Address: 10.64.0.129 State: OK Date/Time: Wed Aug 5 16:16:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1070/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1070 Address: 10.64.0.74 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1085/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1085 Address: 10.64.16.63 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1081/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1081 Address: 10.64.169.12 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1102/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1102 Address: 10.64.136.6 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1078/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1078 Address: 10.64.16.52 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1072/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1072 Address: 10.64.0.77 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1100/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1100 Address: 10.64.136.4 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1077/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1077 Address: 10.64.16.51 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1088/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1088 Address: 10.64.175.9 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1090/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1090 Address: 10.64.130.8 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1098/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1098 Address: 10.64.135.5 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1122/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1122 Address: 10.64.187.9 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1118/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1118 Address: 10.64.169.16 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1119/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1119 Address: 10.64.169.17 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1069/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1069 Address: 10.64.0.73 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1071/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1071 Address: 10.64.0.76 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1082/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1082 Address: 10.64.175.4 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 84, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1089/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1089 Address: 10.64.130.7 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 134, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1087/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1087 Address: 10.64.175.8 State: OK Date/Time: Wed Aug 5 16:16:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3857, relocating_shards: 0, initializing_shards: 17, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 67, active_shards_percent_as_number: 99.56117707795559

1 0

** RECOVERY alert - cirrussearch1080/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1080 Address: 10.64.169.8 State: OK Date/Time: Wed Aug 5 16:16:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3851, relocating_shards: 0, initializing_shards: 23, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.40629839958699

1 0

** RECOVERY alert - cirrussearch1079/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1079 Address: 10.64.16.53 State: OK Date/Time: Wed Aug 5 16:16:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3851, relocating_shards: 0, initializing_shards: 23, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.40629839958699

1 0

** RECOVERY alert - cirrussearch1086/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1086 Address: 10.64.16.64 State: OK Date/Time: Wed Aug 5 16:16:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3851, relocating_shards: 0, initializing_shards: 23, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.40629839958699

1 0

** RECOVERY alert - cirrussearch1094/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1094 Address: 10.64.132.4 State: OK Date/Time: Wed Aug 5 16:16:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3851, relocating_shards: 0, initializing_shards: 23, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.40629839958699

1 0

** RECOVERY alert - cirrussearch1093/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1093 Address: 10.64.132.2 State: OK Date/Time: Wed Aug 5 16:16:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3851, relocating_shards: 0, initializing_shards: 23, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.40629839958699

1 0

** RECOVERY alert - cirrussearch1073/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1073 Address: 10.64.0.78 State: OK Date/Time: Wed Aug 5 16:16:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3850, relocating_shards: 0, initializing_shards: 24, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.38048528652556

1 0

** RECOVERY alert - cirrussearch1068/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1068 Address: 10.64.0.72 State: OK Date/Time: Wed Aug 5 16:16:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3851, relocating_shards: 0, initializing_shards: 23, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.40629839958699

1 0

** RECOVERY alert - cirrussearch1095/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1095 Address: 10.64.132.5 State: OK Date/Time: Wed Aug 5 16:16:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 3850, relocating_shards: 0, initializing_shards: 24, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.38048528652556

1 0

[FIRING:2] CirrusSearchNodeIndexingNotIncreasing elasticsearch search-platform (true true true false production-search-eqiad elasticsearch ops warning eqiad prometheus)
by sre-observability＠wikimedia.org 05 Aug '26

05 Aug '26

1 0

** RECOVERY alert - cirrussearch1095/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1095 Address: 10.64.132.5 State: OK Date/Time: Wed Aug 5 15:48:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9643 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9643 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Wed Aug 5 15:47:37 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1115/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1115 Address: 10.64.16.136 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1123/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1123 Address: 10.64.158.2 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1116/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1116 Address: 10.64.16.138 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1117/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1117 Address: 10.64.16.139 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1121/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1121 Address: 10.64.187.5 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1101/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1101 Address: 10.64.136.5 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1111/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1111 Address: 10.64.0.152 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1097/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1097 Address: 10.64.134.7 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1090/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1090 Address: 10.64.130.8 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1088/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1088 Address: 10.64.175.9 State: OK Date/Time: Wed Aug 5 15:47:35 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 4859, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 347, delayed_unassigned_shards: 347, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.33461390703035

1 0

** RECOVERY alert - cirrussearch1086/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1086 Address: 10.64.16.64 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 44, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1641, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1084/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1084 Address: 10.64.0.129 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 44, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1688, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1108/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1108 Address: 10.64.130.24 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 44, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1669, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1118/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1118 Address: 10.64.169.16 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 44, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1646, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1102/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1102 Address: 10.64.136.6 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 44, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1651, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1072/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1072 Address: 10.64.0.77 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 43, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1611, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1110/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1110 Address: 10.64.134.22 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 43, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1608, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1069/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1069 Address: 10.64.0.73 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 43, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1598, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1081/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1081 Address: 10.64.169.12 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 43, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1603, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1087/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1087 Address: 10.64.175.8 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 41, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1490, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1122/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1122 Address: 10.64.187.9 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 41, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1456, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1085/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1085 Address: 10.64.16.63 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 41, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1384, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1083/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1083 Address: 10.64.175.7 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 41, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1430, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1099/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1099 Address: 10.64.135.6 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 40, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1344, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1079/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1079 Address: 10.64.16.53 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 40, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1309, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1092/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1092 Address: 10.64.131.6 State: OK Date/Time: Wed Aug 5 15:47:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 38, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1231, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1073/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1073 Address: 10.64.0.78 State: OK Date/Time: Wed Aug 5 15:47:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 38, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1215, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch1078/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch1078 Address: 10.64.16.52 State: OK Date/Time: Wed Aug 5 15:47:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-eqiad: cluster_name: production-search-psi-eqiad, status: green, timed_out: False, number_of_nodes: 30, number_of_data_nodes: 30, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1741, active_shards: 5206, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 38, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1180, active_shards_percent_as_number: 100.0

1 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9243 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Wed Aug 5 15:35:33 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.2 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1121/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1121 Address: 10.64.187.5 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1115/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1115 Address: 10.64.16.136 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1111/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1111 Address: 10.64.0.152 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1114/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1114 Address: 10.64.16.133 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1120/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1120 Address: 10.64.48.29 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1123/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1123 Address: 10.64.158.2 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1124/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1124 Address: 10.64.166.2 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1112/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1112 Address: 10.64.0.175 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1117/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1117 Address: 10.64.16.139 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1116/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1116 Address: 10.64.16.138 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1109/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1109 Address: 10.64.152.2 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1091/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1091 Address: 10.64.131.4 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1097/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1097 Address: 10.64.134.7 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1113/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1113 Address: 10.64.16.37 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1101/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1101 Address: 10.64.136.5 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1107/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1107 Address: 10.64.160.2 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1103/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1103 Address: 10.64.183.7 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1125/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1125 Address: 10.64.166.3 State: CRITICAL Date/Time: Wed Aug 5 15:35:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1100/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1100 Address: 10.64.136.4 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1110/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1110 Address: 10.64.134.22 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1099/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1099 Address: 10.64.135.6 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1078/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1078 Address: 10.64.16.52 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1108/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1108 Address: 10.64.130.24 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1084/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1084 Address: 10.64.0.129 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1082/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1082 Address: 10.64.175.4 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1081/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1081 Address: 10.64.169.12 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1122/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1122 Address: 10.64.187.9 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1096/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1096 Address: 10.64.134.6 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1083/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1083 Address: 10.64.175.7 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1090/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1090 Address: 10.64.130.8 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1088/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1088 Address: 10.64.175.9 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1077/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1077 Address: 10.64.16.51 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1087/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1087 Address: 10.64.175.8 State: CRITICAL Date/Time: Wed Aug 5 15:35:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1072/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1072 Address: 10.64.0.77 State: CRITICAL Date/Time: Wed Aug 5 15:35:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1092/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1092 Address: 10.64.131.6 State: CRITICAL Date/Time: Wed Aug 5 15:35:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1089/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1089 Address: 10.64.130.7 State: CRITICAL Date/Time: Wed Aug 5 15:35:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1071/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1071 Address: 10.64.0.76 State: CRITICAL Date/Time: Wed Aug 5 15:35:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1070/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1070 Address: 10.64.0.74 State: CRITICAL Date/Time: Wed Aug 5 15:35:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1085/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1085 Address: 10.64.16.63 State: CRITICAL Date/Time: Wed Aug 5 15:35:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1069/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1069 Address: 10.64.0.73 State: CRITICAL Date/Time: Wed Aug 5 15:35:25 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1080/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1080 Address: 10.64.169.8 State: CRITICAL Date/Time: Wed Aug 5 15:35:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1079/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1079 Address: 10.64.16.53 State: CRITICAL Date/Time: Wed Aug 5 15:35:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1094/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1094 Address: 10.64.132.4 State: CRITICAL Date/Time: Wed Aug 5 15:35:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1086/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1086 Address: 10.64.16.64 State: CRITICAL Date/Time: Wed Aug 5 15:35:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** PROBLEM alert - cirrussearch1068/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1068 Address: 10.64.0.72 State: CRITICAL Date/Time: Wed Aug 5 15:35:23 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 2580 threshold =0.15 breach: cluster_name: production-search-eqiad, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1294, active_shards: 1294, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 2580, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 33.402168301497156

1 0

** RECOVERY alert - cirrussearch1111/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1111 Address: 10.64.0.152 State: OK Date/Time: Wed Aug 5 15:32:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 96, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9403, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1120/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1120 Address: 10.64.48.29 State: OK Date/Time: Wed Aug 5 15:32:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 99, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9789, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1110/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1110 Address: 10.64.134.22 State: OK Date/Time: Wed Aug 5 15:32:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 95, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9354, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1103/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1103 Address: 10.64.183.7 State: OK Date/Time: Wed Aug 5 15:32:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 95, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9270, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1124/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1124 Address: 10.64.166.2 State: OK Date/Time: Wed Aug 5 15:32:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 95, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9341, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1108/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1108 Address: 10.64.130.24 State: OK Date/Time: Wed Aug 5 15:32:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 91, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9150, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9243 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Wed Aug 5 15:32:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 90, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9147, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1115/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1115 Address: 10.64.16.136 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 88, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9008, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1125/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1125 Address: 10.64.166.3 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 89, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9128, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1121/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1121 Address: 10.64.187.5 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 89, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 9146, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1112/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1112 Address: 10.64.0.175 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 88, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8993, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1123/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1123 Address: 10.64.158.2 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 87, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8909, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1113/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1113 Address: 10.64.16.37 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 87, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8925, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1107/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1107 Address: 10.64.160.2 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 86, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8776, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1090/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1090 Address: 10.64.130.8 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 86, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8604, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1091/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1091 Address: 10.64.131.4 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 87, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8814, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1109/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1109 Address: 10.64.152.2 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 86, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8617, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1083/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1083 Address: 10.64.175.7 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 85, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8550, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1070/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1070 Address: 10.64.0.74 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 84, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8512, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1071/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1071 Address: 10.64.0.76 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 81, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8482, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1084/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1084 Address: 10.64.0.129 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 81, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8486, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1072/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1072 Address: 10.64.0.77 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 83, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8499, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1099/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1099 Address: 10.64.135.6 State: OK Date/Time: Wed Aug 5 15:32:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 79, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8480, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1114/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1114 Address: 10.64.16.133 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 76, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8437, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1117/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1117 Address: 10.64.16.139 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 77, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8459, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1096/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1096 Address: 10.64.134.6 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 76, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8426, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1101/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1101 Address: 10.64.136.5 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 75, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8409, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1116/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1116 Address: 10.64.16.138 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 74, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8381, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1087/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1087 Address: 10.64.175.8 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 68, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8303, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1082/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1082 Address: 10.64.175.4 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 75, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8385, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1097/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1097 Address: 10.64.134.7 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 72, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8323, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1088/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1088 Address: 10.64.175.9 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 68, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8215, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1089/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1089 Address: 10.64.130.7 State: OK Date/Time: Wed Aug 5 15:32:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 68, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8235, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1080/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1080 Address: 10.64.169.8 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 67, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8193, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1078/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1078 Address: 10.64.16.52 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 66, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8129, active_shards_percent_as_number: NaN

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts August 2026