August 2026 - Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - cirrussearch1069/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1069 Address: 10.64.0.73 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 65, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8078, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1086/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1086 Address: 10.64.16.64 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 66, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8117, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1079/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1079 Address: 10.64.16.53 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 64, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8045, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1092/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1092 Address: 10.64.131.6 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 65, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8062, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1085/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1085 Address: 10.64.16.63 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 64, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 7937, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1077/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1077 Address: 10.64.16.51 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 64, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 7911, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1122/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1122 Address: 10.64.187.9 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 64, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 7461, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1068/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1068 Address: 10.64.0.72 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 64, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 7397, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1094/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1094 Address: 10.64.132.4 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 64, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 6411, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1081/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1081 Address: 10.64.169.12 State: OK Date/Time: Wed Aug 5 15:32:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 64, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 6411, active_shards_percent_as_number: NaN

1 0

** RECOVERY alert - cirrussearch1100/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch1100 Address: 10.64.136.4 State: OK Date/Time: Wed Aug 5 15:32:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 64, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 6410, active_shards_percent_as_number: NaN

1 0

[FIRING:1] CirrusStreamingUpdaterRateTooLow search-platform (flink-app-consumer-search k8s critical eqiad prometheus)
by sre-observability＠wikimedia.org 05 Aug '26

05 Aug '26

1 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9243 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Wed Aug 5 15:19:36 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch https://search.svc.eqiad.wmnet:9243/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.eqiad.wmnet, port=9243): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9643 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Wed Aug 5 15:19:36 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch https://search.svc.eqiad.wmnet:9643/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.eqiad.wmnet, port=9643): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1113/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1113 Address: 10.64.16.37 State: CRITICAL Date/Time: Wed Aug 5 15:19:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1125/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1125 Address: 10.64.166.3 State: CRITICAL Date/Time: Wed Aug 5 15:19:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1123/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1123 Address: 10.64.158.2 State: CRITICAL Date/Time: Wed Aug 5 15:19:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1121/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1121 Address: 10.64.187.5 State: CRITICAL Date/Time: Wed Aug 5 15:19:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1111/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1111 Address: 10.64.0.152 State: CRITICAL Date/Time: Wed Aug 5 15:19:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1115/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1115 Address: 10.64.16.136 State: CRITICAL Date/Time: Wed Aug 5 15:19:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1120/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1120 Address: 10.64.48.29 State: CRITICAL Date/Time: Wed Aug 5 15:19:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1112/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1112 Address: 10.64.0.175 State: CRITICAL Date/Time: Wed Aug 5 15:19:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1121/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1121 Address: 10.64.187.5 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1116/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1116 Address: 10.64.16.138 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1124/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1124 Address: 10.64.166.2 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1115/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1115 Address: 10.64.16.136 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1109/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1109 Address: 10.64.152.2 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1123/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1123 Address: 10.64.158.2 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1108/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1108 Address: 10.64.130.24 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1107/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1107 Address: 10.64.160.2 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1101/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1101 Address: 10.64.136.5 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1079/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1079 Address: 10.64.16.53 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1091/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1091 Address: 10.64.131.4 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1117/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1117 Address: 10.64.16.139 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1099/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1099 Address: 10.64.135.6 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1103/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1103 Address: 10.64.183.7 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1111/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1111 Address: 10.64.0.152 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1110/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1110 Address: 10.64.134.22 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1117/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1117 Address: 10.64.16.139 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1099/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1099 Address: 10.64.135.6 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1110/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1110 Address: 10.64.134.22 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1116/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1116 Address: 10.64.16.138 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1108/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1108 Address: 10.64.130.24 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1090/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1090 Address: 10.64.130.8 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1090/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1090 Address: 10.64.130.8 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1114/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1114 Address: 10.64.16.133 State: CRITICAL Date/Time: Wed Aug 5 15:19:31 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1101/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1101 Address: 10.64.136.5 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1088/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1088 Address: 10.64.175.9 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1084/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1084 Address: 10.64.0.129 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1097/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1097 Address: 10.64.134.7 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1097/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1097 Address: 10.64.134.7 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1087/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1087 Address: 10.64.175.8 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1093/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1093 Address: 10.64.132.2 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1092/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1092 Address: 10.64.131.6 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1084/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1084 Address: 10.64.0.129 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1096/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1096 Address: 10.64.134.6 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1095/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1095 Address: 10.64.132.5 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1118/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1118 Address: 10.64.169.16 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1083/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1083 Address: 10.64.175.7 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1098/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1098 Address: 10.64.135.5 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1119/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1119 Address: 10.64.169.17 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1102/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1102 Address: 10.64.136.6 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1100/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1100 Address: 10.64.136.4 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1102/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1102 Address: 10.64.136.6 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1122/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1122 Address: 10.64.187.9 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1081/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1081 Address: 10.64.169.12 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1122/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1122 Address: 10.64.187.9 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1118/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1118 Address: 10.64.169.16 State: CRITICAL Date/Time: Wed Aug 5 15:19:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1085/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1085 Address: 10.64.16.63 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1078/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1078 Address: 10.64.16.52 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1083/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1083 Address: 10.64.175.7 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1085/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1085 Address: 10.64.16.63 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1069/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1069 Address: 10.64.0.73 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1086/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1086 Address: 10.64.16.64 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1069/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1069 Address: 10.64.0.73 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1070/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1070 Address: 10.64.0.74 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1086/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1086 Address: 10.64.16.64 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1072/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1072 Address: 10.64.0.77 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1078/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1078 Address: 10.64.16.52 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1088/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1088 Address: 10.64.175.9 State: CRITICAL Date/Time: Wed Aug 5 15:19:29 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1072/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1072 Address: 10.64.0.77 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1071/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1071 Address: 10.64.0.76 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1087/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1087 Address: 10.64.175.8 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1068/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1068 Address: 10.64.0.72 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1082/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1082 Address: 10.64.175.4 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1073/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1073 Address: 10.64.0.78 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1080/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1080 Address: 10.64.169.8 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1079/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1079 Address: 10.64.16.53 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1092/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1092 Address: 10.64.131.6 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1077/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1077 Address: 10.64.16.51 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1095/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1095 Address: 10.64.132.5 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1089/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1089 Address: 10.64.130.7 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1094/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1094 Address: 10.64.132.4 State: CRITICAL Date/Time: Wed Aug 5 15:19:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1081/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch1081 Address: 10.64.169.12 State: CRITICAL Date/Time: Wed Aug 5 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch1073/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 05 Aug '26

05 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch1073 Address: 10.64.0.78 State: CRITICAL Date/Time: Wed Aug 5 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - cirrussearch2073/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2073 Address: 10.192.23.25 State: OK Date/Time: Tue Aug 4 16:50:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: green, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 5195, relocating_shards: 2, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9443 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Aug 4 16:23:56 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2112/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2112 Address: 10.192.4.6 State: OK Date/Time: Tue Aug 4 16:23:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2114/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2114 Address: 10.192.36.9 State: OK Date/Time: Tue Aug 4 16:23:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2111/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2111 Address: 10.192.23.16 State: OK Date/Time: Tue Aug 4 16:23:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2092/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2092 Address: 10.192.11.24 State: OK Date/Time: Tue Aug 4 16:23:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2100/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2100 Address: 10.192.28.19 State: OK Date/Time: Tue Aug 4 16:23:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2061/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2061 Address: 10.192.6.19 State: OK Date/Time: Tue Aug 4 16:23:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2081/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2081 Address: 10.192.28.18 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2088/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2088 Address: 10.192.23.23 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2084/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2084 Address: 10.192.39.23 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2070/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2070 Address: 10.192.13.16 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2065/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2065 Address: 10.192.28.14 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2067/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2067 Address: 10.192.42.15 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2099/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2099 Address: 10.192.26.18 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2077/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2077 Address: 10.192.11.19 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2063/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2063 Address: 10.192.11.18 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2098/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2098 Address: 10.192.26.17 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2090/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2090 Address: 10.192.9.19 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2074/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2074 Address: 10.192.23.24 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2106/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2106 Address: 10.192.39.24 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2087/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2087 Address: 10.192.23.22 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2105/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2105 Address: 10.192.37.13 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2104/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2104 Address: 10.192.37.14 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2097/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2097 Address: 10.192.13.18 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2102/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2102 Address: 10.192.31.20 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** RECOVERY alert - cirrussearch2093/OpenSearch health check for shards on 9400 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9400 Host: cirrussearch2093 Address: 10.192.11.25 State: OK Date/Time: Tue Aug 4 16:23:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4810, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 385, delayed_unassigned_shards: 385, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.58902791145333

1 0

** PROBLEM alert - cirrussearch2111/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2111 Address: 10.192.23.16 State: CRITICAL Date/Time: Tue Aug 4 16:18:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2112/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2112 Address: 10.192.4.6 State: CRITICAL Date/Time: Tue Aug 4 16:18:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2114/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2114 Address: 10.192.36.9 State: CRITICAL Date/Time: Tue Aug 4 16:18:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2087/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2087 Address: 10.192.23.22 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2065/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2065 Address: 10.192.28.14 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2067/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2067 Address: 10.192.42.15 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2077/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2077 Address: 10.192.11.19 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2084/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2084 Address: 10.192.39.23 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2074/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2074 Address: 10.192.23.24 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2092/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2092 Address: 10.192.11.24 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2093/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2093 Address: 10.192.11.25 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2073/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2073 Address: 10.192.23.25 State: CRITICAL Date/Time: Tue Aug 4 16:18:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2088/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2088 Address: 10.192.23.23 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2098/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2098 Address: 10.192.26.17 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2081/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2081 Address: 10.192.28.18 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2061/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2061 Address: 10.192.6.19 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2070/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2070 Address: 10.192.13.16 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2063/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2063 Address: 10.192.11.18 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2100/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2100 Address: 10.192.28.19 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2099/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2099 Address: 10.192.26.18 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2102/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2102 Address: 10.192.31.20 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2097/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2097 Address: 10.192.13.18 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2104/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2104 Address: 10.192.37.14 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2106/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2106 Address: 10.192.39.24 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2090/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2090 Address: 10.192.9.19 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2105/OpenSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9400 Host: cirrussearch2105 Address: 10.192.37.13 State: CRITICAL Date/Time: Tue Aug 4 16:18:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9443 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Aug 4 16:18:02 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch https://search.svc.codfw.wmnet:9443/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.codfw.wmnet, port=9443): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - cirrussearch2076/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2076 Address: 10.192.9.22 State: OK Date/Time: Tue Aug 4 16:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Aug 4 15:38:38 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2109/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2109 Address: 10.192.42.16 State: OK Date/Time: Tue Aug 4 15:38:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2107/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2107 Address: 10.192.39.22 State: OK Date/Time: Tue Aug 4 15:38:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2113/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2113 Address: 10.192.29.8 State: OK Date/Time: Tue Aug 4 15:38:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2110/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2110 Address: 10.192.14.12 State: OK Date/Time: Tue Aug 4 15:38:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2071/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2071 Address: 10.192.31.19 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2079/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2079 Address: 10.192.21.30 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2066/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2066 Address: 10.192.28.16 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2075/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2075 Address: 10.192.9.20 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2068/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2068 Address: 10.192.42.13 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2094/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2094 Address: 10.192.11.23 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2101/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2101 Address: 10.192.28.17 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2085/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2085 Address: 10.192.39.21 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2082/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2082 Address: 10.192.28.15 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2072/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2072 Address: 10.192.39.20 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2062/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2062 Address: 10.192.6.17 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2069/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2069 Address: 10.192.9.18 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2078/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2078 Address: 10.192.11.20 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2083/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2083 Address: 10.192.31.21 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2064/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2064 Address: 10.192.11.22 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2103/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2103 Address: 10.192.31.18 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2080/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2080 Address: 10.192.21.31 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2089/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2089 Address: 10.192.6.18 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2108/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2108 Address: 10.192.42.18 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2091/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2091 Address: 10.192.9.21 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2095/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2095 Address: 10.192.13.19 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** RECOVERY alert - cirrussearch2096/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2096 Address: 10.192.13.20 State: OK Date/Time: Tue Aug 4 15:38:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 27, number_of_data_nodes: 27, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4651, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 553, delayed_unassigned_shards: 186, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.37355880092237

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Aug 4 15:35:48 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch https://search.svc.codfw.wmnet:9643/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.codfw.wmnet, port=9643): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Aug 4 15:32:42 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: green, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5204, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 6733, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 979868, active_shards_percent_as_number: 100.0

1 0

** PROBLEM alert - cirrussearch2072/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2072 Address: 10.192.39.20 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2066/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2066 Address: 10.192.28.16 State: CRITICAL Date/Time: Tue Aug 4 15:19:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2094/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2094 Address: 10.192.11.23 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2062/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2062 Address: 10.192.6.17 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2082/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2082 Address: 10.192.28.15 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2075/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2075 Address: 10.192.9.20 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2071/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2071 Address: 10.192.31.19 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2103/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2103 Address: 10.192.31.18 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2101/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2101 Address: 10.192.28.17 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2079/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2079 Address: 10.192.21.30 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2108/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2108 Address: 10.192.42.18 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2076/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2076 Address: 10.192.9.22 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2069/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2069 Address: 10.192.9.18 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2080/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2080 Address: 10.192.21.31 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2078/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2078 Address: 10.192.11.20 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2083/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2083 Address: 10.192.31.21 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2064/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2064 Address: 10.192.11.22 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2085/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2085 Address: 10.192.39.21 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2068/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2068 Address: 10.192.42.13 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2091/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2091 Address: 10.192.9.21 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2089/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2089 Address: 10.192.6.18 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts August 2026