May 2022 - Discovery-alerts - lists.wikimedia.org

** ACKNOWLEDGEMENT alert - relforge1004/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Tue May 24 05:20:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Ryan Kemper: https://phabricator.wikimedia.org/T308770 Additional Info: CRITICAL - elasticsearch inactive shards 16 threshold =0.15 breach: cluster_name: relforge-eqiad-small-alpha, status: red, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 21, active_shards: 26, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 16, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 61.904761904761905

1 year, 11 months

1
0
0 0

** ACKNOWLEDGEMENT alert - relforge1003/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9400 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Tue May 24 05:20:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Ryan Kemper: https://phabricator.wikimedia.org/T308770 Additional Info: CRITICAL - elasticsearch inactive shards 16 threshold =0.15 breach: cluster_name: relforge-eqiad-small-alpha, status: red, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 21, active_shards: 26, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 16, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 61.904761904761905

1 year, 11 months

1
0
0 0

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Tue May 24 05:07:38 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 16 threshold =0.15 breach: cluster_name: relforge-eqiad-small-alpha, status: red, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 21, active_shards: 26, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 16, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 61.904761904761905

1 year, 11 months

1
0
0 0

** RECOVERY alert - relforge1003/ElasticSearch health check for shards on 9200 is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: OK Date/Time: Tue May 24 04:10:26 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: cluster_name: relforge-eqiad, status: red, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 137, active_shards: 274, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 33, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.25081433224756

1 year, 11 months

1
0
0 0

** ACKNOWLEDGEMENT alert - relforge1003/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9400 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Tue May 24 01:26:00 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Ryan Kemper: https://phabricator.wikimedia.org/T308770 Additional Info: CRITICAL - elasticsearch inactive shards 21 threshold =0.15 breach: cluster_name: relforge-eqiad-small-alpha, status: red, timed_out: False, number_of_nodes: 1, number_of_data_nodes: 1, active_primary_shards: 21, active_shards: 21, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 21, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 50.0

1 year, 11 months

1
0
0 0

** ACKNOWLEDGEMENT alert - relforge1003/ElasticSearch health check for shards on 9200 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Tue May 24 01:26:00 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Ryan Kemper: https://phabricator.wikimedia.org/T308770 Additional Info: CRITICAL - elasticsearch inactive shards 167 threshold =0.15 breach: cluster_name: relforge-eqiad, status: red, timed_out: False, number_of_nodes: 1, number_of_data_nodes: 1, active_primary_shards: 137, active_shards: 137, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 167, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 45.06578947368421

1 year, 11 months

1
0
0 0

** PROBLEM alert - relforge1003/ElasticSearch health check for shards on 9200 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Tue May 24 01:14:58 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 167 threshold =0.15 breach: cluster_name: relforge-eqiad, status: red, timed_out: False, number_of_nodes: 1, number_of_data_nodes: 1, active_primary_shards: 137, active_shards: 137, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 167, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 45.06578947368421

1 year, 11 months

1
0
0 0

** PROBLEM alert - relforge1003/ElasticSearch health check for shards on 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Tue May 24 01:13:42 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 21 threshold =0.15 breach: cluster_name: relforge-eqiad-small-alpha, status: red, timed_out: False, number_of_nodes: 1, number_of_data_nodes: 1, active_primary_shards: 21, active_shards: 21, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 21, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 50.0

1 year, 11 months

1
0
0 0

** RECOVERY alert - relforge1004/ElasticSearch health check for shards on 9200 is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: OK Date/Time: Tue May 24 00:43:26 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: cluster_name: relforge-eqiad, active_shards_percent_as_number: 99.6742671009772, active_shards: 306, timed_out: False, delayed_unassigned_shards: 0, unassigned_shards: 0, task_max_waiting_in_queue_millis: 0, number_of_nodes: 2, initializing_shards: 1, active_primary_shards: 170, relocating_shards: 0, status: yellow, number_of_in_flight_fetch: 0, number_of_pending_tasks: 0, number_of_data_nodes: 2

1 year, 11 months

1
0
0 0

** RECOVERY alert - relforge1003/ElasticSearch health check for shards on 9200 is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: OK Date/Time: Tue May 24 00:43:10 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 170, active_shards: 300, relocating_shards: 0, initializing_shards: 2, unassigned_shards: 5, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 97.71986970684038

1 year, 11 months

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts May 2022