- Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - graphite1005/CirrusSearch full_text eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: PROBLEM Service: CirrusSearch full_text eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Sat Jun 22 22:29:51 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 50.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - logstash1027/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1027 Address: 10.64.16.169 State: OK Date/Time: Sat Jun 22 22:29:37 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1281, relocating_shards: 0, initializing_shards: 13, unassigned_shards: 435, delayed_unassigned_shards: 0, number_of_pending_tasks: 10, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1486, active_shards_percent_as_number: 74.08906882591093

1 0

** RECOVERY alert - logstash1023/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1023 Address: 10.64.0.183 State: OK Date/Time: Sat Jun 22 22:29:37 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1281, relocating_shards: 0, initializing_shards: 13, unassigned_shards: 435, delayed_unassigned_shards: 0, number_of_pending_tasks: 8, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 952, active_shards_percent_as_number: 74.08906882591093

1 0

** RECOVERY alert - logstash1030/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1030 Address: 10.64.48.22 State: OK Date/Time: Sat Jun 22 22:29:35 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1281, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 443, delayed_unassigned_shards: 0, number_of_pending_tasks: 6, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 527, active_shards_percent_as_number: 74.08906882591093

1 0

** RECOVERY alert - logging-hd1001/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logging-hd1001 Address: 10.64.0.139 State: OK Date/Time: Sat Jun 22 22:29:35 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1281, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 443, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 24, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 74.08906882591093

1 0

** PROBLEM alert - graphite1005/CirrusSearch comp_suggest eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: PROBLEM Service: CirrusSearch comp_suggest eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Sat Jun 22 22:29:35 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 50.00% of data above the critical threshold [250.0]

1 0

** RECOVERY alert - logging-hd1002/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logging-hd1002 Address: 10.64.16.76 State: OK Date/Time: Sat Jun 22 22:29:35 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1281, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 443, delayed_unassigned_shards: 0, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 211, active_shards_percent_as_number: 74.08906882591093

1 0

** RECOVERY alert - logstash1026/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1026 Address: 10.64.0.197 State: OK Date/Time: Sat Jun 22 22:29:35 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1281, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 443, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 24, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 74.08906882591093

1 0

** RECOVERY alert - logging-hd1003/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logging-hd1003 Address: 10.64.48.124 State: OK Date/Time: Sat Jun 22 22:29:31 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1274, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 450, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 24, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 73.68421052631578

1 0

** RECOVERY alert - logstash1035/OpenSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 22 Jun '24

22 Jun '24

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash1035 Address: 10.64.48.60 State: OK Date/Time: Sat Jun 22 22:29:31 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-eqiad: cluster_name: production-elk7-eqiad, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 731, active_shards: 1273, relocating_shards: 0, initializing_shards: 6, unassigned_shards: 450, delayed_unassigned_shards: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 1622, active_shards_percent_as_number: 73.62637362637363

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts