August 2026 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - cirrussearch2096/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2096 Address: 10.192.13.20 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2095/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2095 Address: 10.192.13.19 State: CRITICAL Date/Time: Tue Aug 4 15:19:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Aug 4 15:18:44 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch https://search.svc.codfw.wmnet:9643/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.codfw.wmnet, port=9643): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2113/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2113 Address: 10.192.29.8 State: CRITICAL Date/Time: Tue Aug 4 15:18:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2109/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2109 Address: 10.192.42.16 State: CRITICAL Date/Time: Tue Aug 4 15:18:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2107/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2107 Address: 10.192.39.22 State: CRITICAL Date/Time: Tue Aug 4 15:18:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2110/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2110 Address: 10.192.14.12 State: CRITICAL Date/Time: Tue Aug 4 15:18:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - cirrussearch2084/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2084 Address: 10.192.39.23 State: OK Date/Time: Tue Aug 4 15:14:24 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: green, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3761, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch2114/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2114 Address: 10.192.36.9 State: OK Date/Time: Tue Aug 4 15:14:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: green, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3761, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch2111/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2111 Address: 10.192.23.16 State: OK Date/Time: Tue Aug 4 15:14:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: green, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3761, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch2112/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2112 Address: 10.192.4.6 State: OK Date/Time: Tue Aug 4 15:14:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: green, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3761, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch2115/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2115 Address: 10.192.38.8 State: OK Date/Time: Tue Aug 4 15:14:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: green, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3761, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch2113/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2113 Address: 10.192.29.8 State: OK Date/Time: Tue Aug 4 15:14:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: green, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3761, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch2110/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2110 Address: 10.192.14.12 State: OK Date/Time: Tue Aug 4 15:14:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: green, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3761, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch2109/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2109 Address: 10.192.42.16 State: OK Date/Time: Tue Aug 4 15:14:22 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: green, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3761, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - cirrussearch2101/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2101 Address: 10.192.28.17 State: OK Date/Time: Tue Aug 4 15:14:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 354, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2077/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2077 Address: 10.192.11.19 State: OK Date/Time: Tue Aug 4 15:14:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 368, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2088/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2088 Address: 10.192.23.23 State: OK Date/Time: Tue Aug 4 15:14:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 352, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2073/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2073 Address: 10.192.23.25 State: OK Date/Time: Tue Aug 4 15:14:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 351, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2100/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2100 Address: 10.192.28.19 State: OK Date/Time: Tue Aug 4 15:14:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 344, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2094/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2094 Address: 10.192.11.23 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 339, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2085/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2085 Address: 10.192.39.21 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 343, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2066/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2066 Address: 10.192.28.16 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 330, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2075/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2075 Address: 10.192.9.20 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 331, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2087/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2087 Address: 10.192.23.22 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 339, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2072/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2072 Address: 10.192.39.20 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 335, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2068/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2068 Address: 10.192.42.13 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 331, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2069/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2069 Address: 10.192.9.18 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 305, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2071/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2071 Address: 10.192.31.19 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 308, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2063/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2063 Address: 10.192.11.18 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 335, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2082/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2082 Address: 10.192.28.15 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 328, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2092/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2092 Address: 10.192.11.24 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 326, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2065/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2065 Address: 10.192.28.14 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 319, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2070/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2070 Address: 10.192.13.16 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 294, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2074/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2074 Address: 10.192.23.24 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 328, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2081/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2081 Address: 10.192.28.18 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 336, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2076/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2076 Address: 10.192.9.22 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 301, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2086/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2086 Address: 10.192.42.17 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 321, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2078/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2078 Address: 10.192.11.20 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 302, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2091/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2091 Address: 10.192.9.21 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 310, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2093/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2093 Address: 10.192.11.25 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 314, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2079/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2079 Address: 10.192.21.30 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 300, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2080/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2080 Address: 10.192.21.31 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 23, delayed_unassigned_shards: 23, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 371, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2106/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2106 Address: 10.192.39.24 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 294, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2062/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2062 Address: 10.192.6.17 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 308, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2064/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2064 Address: 10.192.11.22 State: OK Date/Time: Tue Aug 4 15:14:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 293, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2102/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2102 Address: 10.192.31.20 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 292, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2108/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2108 Address: 10.192.42.18 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 23, delayed_unassigned_shards: 23, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 365, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2098/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2098 Address: 10.192.26.17 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 23, delayed_unassigned_shards: 23, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 341, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2067/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2067 Address: 10.192.42.15 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 23, delayed_unassigned_shards: 23, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 370, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2103/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2103 Address: 10.192.31.18 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 23, delayed_unassigned_shards: 23, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 352, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2107/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2107 Address: 10.192.39.22 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 22, delayed_unassigned_shards: 22, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 294, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2104/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2104 Address: 10.192.37.14 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 23, delayed_unassigned_shards: 23, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 340, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2105/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2105 Address: 10.192.37.13 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 23, delayed_unassigned_shards: 23, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 333, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2083/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2083 Address: 10.192.31.21 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3733, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 28, delayed_unassigned_shards: 28, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.25551714969423

1 0

** RECOVERY alert - cirrussearch2099/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2099 Address: 10.192.26.18 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3734, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 23, delayed_unassigned_shards: 23, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 322, active_shards_percent_as_number: 99.28210582291943

1 0

** RECOVERY alert - cirrussearch2096/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2096 Address: 10.192.13.20 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3733, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 28, delayed_unassigned_shards: 28, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.25551714969423

1 0

** RECOVERY alert - cirrussearch2090/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2090 Address: 10.192.9.19 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3733, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 28, delayed_unassigned_shards: 28, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.25551714969423

1 0

** RECOVERY alert - cirrussearch2089/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2089 Address: 10.192.6.18 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3733, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 28, delayed_unassigned_shards: 28, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.25551714969423

1 0

** RECOVERY alert - cirrussearch2095/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2095 Address: 10.192.13.19 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3733, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 28, delayed_unassigned_shards: 28, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.25551714969423

1 0

** RECOVERY alert - cirrussearch2097/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: cirrussearch2097 Address: 10.192.13.18 State: OK Date/Time: Tue Aug 4 15:14:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 55, number_of_data_nodes: 55, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1263, active_shards: 3733, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 28, delayed_unassigned_shards: 28, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 99.25551714969423

1 0

** PROBLEM alert - cirrussearch2084/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2084 Address: 10.192.39.23 State: CRITICAL Date/Time: Tue Aug 4 15:13:32 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2114/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2114 Address: 10.192.36.9 State: CRITICAL Date/Time: Tue Aug 4 15:13:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2111/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2111 Address: 10.192.23.16 State: CRITICAL Date/Time: Tue Aug 4 15:13:30 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2094/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2094 Address: 10.192.11.23 State: CRITICAL Date/Time: Tue Aug 4 15:13:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2115/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2115 Address: 10.192.38.8 State: CRITICAL Date/Time: Tue Aug 4 15:13:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2110/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2110 Address: 10.192.14.12 State: CRITICAL Date/Time: Tue Aug 4 15:13:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2113/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2113 Address: 10.192.29.8 State: CRITICAL Date/Time: Tue Aug 4 15:13:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2112/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2112 Address: 10.192.4.6 State: CRITICAL Date/Time: Tue Aug 4 15:13:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2109/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2109 Address: 10.192.42.16 State: CRITICAL Date/Time: Tue Aug 4 15:13:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2066/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2066 Address: 10.192.28.16 State: CRITICAL Date/Time: Tue Aug 4 15:13:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2100/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2100 Address: 10.192.28.19 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2073/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2073 Address: 10.192.23.25 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2101/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2101 Address: 10.192.28.17 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2087/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2087 Address: 10.192.23.22 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2072/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2072 Address: 10.192.39.20 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2069/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2069 Address: 10.192.9.18 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2088/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2088 Address: 10.192.23.23 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2092/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2092 Address: 10.192.11.24 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2071/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2071 Address: 10.192.31.19 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2082/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2082 Address: 10.192.28.15 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2063/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2063 Address: 10.192.11.18 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2075/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2075 Address: 10.192.9.20 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2068/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2068 Address: 10.192.42.13 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2067/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2067 Address: 10.192.42.15 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2076/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2076 Address: 10.192.9.22 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2077/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2077 Address: 10.192.11.19 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2080/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2080 Address: 10.192.21.31 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2079/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2079 Address: 10.192.21.30 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2085/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2085 Address: 10.192.39.21 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2062/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2062 Address: 10.192.6.17 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2070/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2070 Address: 10.192.13.16 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2086/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2086 Address: 10.192.42.17 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2081/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2081 Address: 10.192.28.18 State: CRITICAL Date/Time: Tue Aug 4 15:13:27 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2093/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2093 Address: 10.192.11.25 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2107/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2107 Address: 10.192.39.22 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2074/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2074 Address: 10.192.23.24 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2078/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2078 Address: 10.192.11.20 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2105/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2105 Address: 10.192.37.13 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2102/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2102 Address: 10.192.31.20 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2103/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2103 Address: 10.192.31.18 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2098/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2098 Address: 10.192.26.17 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2108/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2108 Address: 10.192.42.18 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2091/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2091 Address: 10.192.9.21 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2104/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2104 Address: 10.192.37.14 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2106/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2106 Address: 10.192.39.24 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2099/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2099 Address: 10.192.26.18 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2065/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2065 Address: 10.192.28.14 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2064/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2064 Address: 10.192.11.22 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2083/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2083 Address: 10.192.31.21 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2097/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2097 Address: 10.192.13.18 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2090/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2090 Address: 10.192.9.19 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2096/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2096 Address: 10.192.13.20 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2089/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2089 Address: 10.192.6.18 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2095/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: cirrussearch2095 Address: 10.192.13.19 State: CRITICAL Date/Time: Tue Aug 4 15:13:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Aug 4 15:00:34 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5188, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 12, delayed_unassigned_shards: 12, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 100, active_shards_percent_as_number: 99.69254419677172

1 0

** RECOVERY alert - cirrussearch2107/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2107 Address: 10.192.39.22 State: OK Date/Time: Tue Aug 4 15:00:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5050, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 149, delayed_unassigned_shards: 149, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 39, active_shards_percent_as_number: 97.04073789392775

1 0

** RECOVERY alert - cirrussearch2113/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2113 Address: 10.192.29.8 State: OK Date/Time: Tue Aug 4 15:00:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5050, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 149, delayed_unassigned_shards: 149, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 42, active_shards_percent_as_number: 97.04073789392775

1 0

** RECOVERY alert - cirrussearch2110/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2110 Address: 10.192.14.12 State: OK Date/Time: Tue Aug 4 15:00:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5050, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 149, delayed_unassigned_shards: 149, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 15, active_shards_percent_as_number: 97.04073789392775

1 0

** RECOVERY alert - cirrussearch2109/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2109 Address: 10.192.42.16 State: OK Date/Time: Tue Aug 4 15:00:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5050, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 149, delayed_unassigned_shards: 149, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 42, active_shards_percent_as_number: 97.04073789392775

1 0

** RECOVERY alert - cirrussearch2115/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2115 Address: 10.192.38.8 State: OK Date/Time: Tue Aug 4 15:00:20 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5050, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 149, delayed_unassigned_shards: 149, number_of_pending_tasks: 5, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 52, active_shards_percent_as_number: 97.04073789392775

1 0

** RECOVERY alert - cirrussearch2071/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2071 Address: 10.192.31.19 State: OK Date/Time: Tue Aug 4 15:00:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5037, relocating_shards: 0, initializing_shards: 1, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 96.79093005380477

1 0

** RECOVERY alert - cirrussearch2101/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2101 Address: 10.192.28.17 State: OK Date/Time: Tue Aug 4 15:00:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 78, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2075/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2075 Address: 10.192.9.20 State: OK Date/Time: Tue Aug 4 15:00:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 78, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2094/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2094 Address: 10.192.11.23 State: OK Date/Time: Tue Aug 4 15:00:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 78, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2085/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2085 Address: 10.192.39.21 State: OK Date/Time: Tue Aug 4 15:00:19 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 82, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2064/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2064 Address: 10.192.11.22 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 75, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2066/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2066 Address: 10.192.28.16 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 74, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2068/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2068 Address: 10.192.42.13 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 62, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2069/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2069 Address: 10.192.9.18 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 78, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2078/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2078 Address: 10.192.11.20 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 74, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2076/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2076 Address: 10.192.9.22 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 67, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2072/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2072 Address: 10.192.39.20 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 67, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2079/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2079 Address: 10.192.21.30 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 53, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2103/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2103 Address: 10.192.31.18 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5033, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 33, active_shards_percent_as_number: 96.71406610299769

1 0

** RECOVERY alert - cirrussearch2062/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2062 Address: 10.192.6.17 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5034, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 7, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 49, active_shards_percent_as_number: 96.73328209069946

1 0

** RECOVERY alert - cirrussearch2082/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2082 Address: 10.192.28.15 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5029, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 170, delayed_unassigned_shards: 170, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 40, active_shards_percent_as_number: 96.63720215219062

1 0

** RECOVERY alert - cirrussearch2108/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2108 Address: 10.192.42.18 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5033, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 166, delayed_unassigned_shards: 166, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 33, active_shards_percent_as_number: 96.71406610299769

1 0

** RECOVERY alert - cirrussearch2080/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2080 Address: 10.192.21.31 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5029, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 170, delayed_unassigned_shards: 170, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 39, active_shards_percent_as_number: 96.63720215219062

1 0

** RECOVERY alert - cirrussearch2091/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2091 Address: 10.192.9.21 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5029, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 170, delayed_unassigned_shards: 170, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 14, active_shards_percent_as_number: 96.63720215219062

1 0

** RECOVERY alert - cirrussearch2089/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2089 Address: 10.192.6.18 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5029, relocating_shards: 0, initializing_shards: 3, unassigned_shards: 172, delayed_unassigned_shards: 172, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 119, active_shards_percent_as_number: 96.63720215219062

1 0

** RECOVERY alert - cirrussearch2096/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2096 Address: 10.192.13.20 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5029, relocating_shards: 0, initializing_shards: 3, unassigned_shards: 172, delayed_unassigned_shards: 172, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 117, active_shards_percent_as_number: 96.63720215219062

1 0

** RECOVERY alert - cirrussearch2095/OpenSearch health check for shards on 9600 is OK **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9600 Host: cirrussearch2095 Address: 10.192.13.19 State: OK Date/Time: Tue Aug 4 15:00:18 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 28, number_of_data_nodes: 28, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 5029, relocating_shards: 0, initializing_shards: 3, unassigned_shards: 172, delayed_unassigned_shards: 172, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 103, active_shards_percent_as_number: 96.63720215219062

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Aug 4 14:57:42 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch https://search.svc.codfw.wmnet:9643/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.codfw.wmnet, port=9643): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2109/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2109 Address: 10.192.42.16 State: CRITICAL Date/Time: Tue Aug 4 14:57:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2113/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2113 Address: 10.192.29.8 State: CRITICAL Date/Time: Tue Aug 4 14:57:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2107/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2107 Address: 10.192.39.22 State: CRITICAL Date/Time: Tue Aug 4 14:57:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2115/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2115 Address: 10.192.38.8 State: CRITICAL Date/Time: Tue Aug 4 14:57:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2110/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2110 Address: 10.192.14.12 State: CRITICAL Date/Time: Tue Aug 4 14:57:28 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2076/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2076 Address: 10.192.9.22 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2085/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2085 Address: 10.192.39.21 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2068/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2068 Address: 10.192.42.13 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2101/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2101 Address: 10.192.28.17 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2069/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2069 Address: 10.192.9.18 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2072/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2072 Address: 10.192.39.20 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2066/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2066 Address: 10.192.28.16 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2078/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2078 Address: 10.192.11.20 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2064/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2064 Address: 10.192.11.22 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2071/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2071 Address: 10.192.31.19 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2075/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2075 Address: 10.192.9.20 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2079/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2079 Address: 10.192.21.30 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2094/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2094 Address: 10.192.11.23 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2062/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2062 Address: 10.192.6.17 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2108/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2108 Address: 10.192.42.18 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2082/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2082 Address: 10.192.28.15 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2103/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2103 Address: 10.192.31.18 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2080/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2080 Address: 10.192.21.31 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2091/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2091 Address: 10.192.9.21 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2095/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2095 Address: 10.192.13.19 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2096/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2096 Address: 10.192.13.20 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - cirrussearch2089/OpenSearch health check for shards on 9600 is CRITICAL **
by nagios＠alert1002.wikimedia.org 04 Aug '26

04 Aug '26

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9600 Host: cirrussearch2089 Address: 10.192.6.18 State: CRITICAL Date/Time: Tue Aug 4 14:57:26 UTC 2026 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4)

1 0

Airflow alert: <TaskInstance: embeddings_weekly.wait_for_structured_content_snapshots scheduled__2026-07-26T13:00:00+00:00 [failed]>
by noreply＠wikimedia.org 03 Aug '26

03 Aug '26

1 0

Airflow alert: <TaskInstance: import_enterprise_snapshots_weekly.import scheduled__2026-07-26T13:00:00+00:00 [failed]>
by noreply＠wikimedia.org 03 Aug '26

03 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 03 Aug '26

03 Aug '26

1 0

Airflow alert: <TaskInstance: mjolnir_weekly.train-nowiki-dbn-20180215-query_explorer-pruned_mrmr scheduled__2026-07-23T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

Airflow alert: <TaskInstance: import_enterprise_snapshots_weekly.import scheduled__2026-07-26T13:00:00+00:00 [failed]>
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

Airflow alert: <TaskInstance: import_enterprise_snapshots_weekly.import scheduled__2026-07-26T13:00:00+00:00 [failed]>
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

[airflow] SLA miss on DAG=wcqs_streaming_updater_reconcile_hourly
by noreply＠wikimedia.org 02 Aug '26

02 Aug '26

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts August 2026