July 2022 - Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - cloudelastic1006/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 07 Jul '22

07 Jul '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1006 Address: 208.80.154.138 State: OK Date/Time: Thu Jul 7 17:26:15 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, active_primary_shards: 727, active_shards: 1167, relocating_shards: 0, initializing_shards: 2, unassigned_shards: 288, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 80.09608785175017

1 0

** RECOVERY alert - cloudelastic1005/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 07 Jul '22

07 Jul '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1005 Address: 208.80.154.9 State: OK Date/Time: Thu Jul 7 17:25:41 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: status: yellow, number_of_in_flight_fetch: 0, active_primary_shards: 727, delayed_unassigned_shards: 0, active_shards_percent_as_number: 80.09608785175017, relocating_shards: 0, number_of_nodes: 6, number_of_pending_tasks: 0, number_of_data_nodes: 6, task_max_waiting_in_queue_millis: 0, timed_out: False, active_shards: 1167, initializing_shards: 2, cluster_name: cloudelastic-chi-eqiad, unassigned_shards: 288

1 0

** RECOVERY alert - cloudelastic1001/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 07 Jul '22

07 Jul '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: OK Date/Time: Thu Jul 7 17:25:05 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: status: yellow, timed_out: False, cluster_name: cloudelastic-chi-eqiad, active_shards_percent_as_number: 80.02745367192861, unassigned_shards: 289, number_of_pending_tasks: 1, initializing_shards: 2, number_of_data_nodes: 6, delayed_unassigned_shards: 0, relocating_shards: 0, number_of_in_flight_fetch: 0, active_primary_shards: 727, number_of_nodes: 6, task_max_waiting_in_queue_millis: 0, active_shards: 1166

1 0

** PROBLEM alert - cloudelastic1001/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jul '22

07 Jul '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: CRITICAL Date/Time: Thu Jul 7 17:22:57 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 292 threshold =0.2 breach: relocating_shards: 0, initializing_shards: 0, status: yellow, timed_out: False, number_of_data_nodes: 6, number_of_pending_tasks: 0, delayed_unassigned_shards: 0, task_max_waiting_in_queue_millis: 0, active_shards: 1165, active_shards_percent_as_number: 79.95881949210707, active_primary_shards: 727, number_of_in_flight_fetch: 0, unassigned_shards: 292, cluster_name: cloudelastic-chi-eqiad, number_of_nodes: 6

1 0

** PROBLEM alert - cloudelastic1005/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jul '22

07 Jul '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1005 Address: 208.80.154.9 State: CRITICAL Date/Time: Thu Jul 7 17:09:05 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 292 threshold =0.2 breach: cluster_name: cloudelastic-chi-eqiad, active_primary_shards: 727, initializing_shards: 0, number_of_data_nodes: 5, number_of_in_flight_fetch: 0, unassigned_shards: 292, active_shards: 1165, number_of_nodes: 5, status: yellow, timed_out: False, delayed_unassigned_shards: 0, relocating_shards: 0, number_of_pending_tasks: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 79.95881949210707

1 0

** PROBLEM alert - cloudelastic1003/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jul '22

07 Jul '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1003 Address: 208.80.154.83 State: CRITICAL Date/Time: Thu Jul 7 16:57:36 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 292 threshold =0.2 breach: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, active_primary_shards: 727, active_shards: 1165, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 292, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 79.95881949210707

1 0

** PROBLEM alert - cloudelastic1004/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jul '22

07 Jul '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1004 Address: 208.80.155.107 State: CRITICAL Date/Time: Thu Jul 7 16:54:52 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 292 threshold =0.2 breach: active_primary_shards: 727, number_of_in_flight_fetch: 0, timed_out: False, delayed_unassigned_shards: 0, initializing_shards: 0, number_of_nodes: 5, unassigned_shards: 292, number_of_pending_tasks: 0, status: yellow, number_of_data_nodes: 5, relocating_shards: 0, active_shards_percent_as_number: 79.95881949210707, cluster_name: cloudelastic-chi-eqiad, active_shards: 1165, task_max_waiting_in_queue_millis: 0

1 0

** PROBLEM alert - cloudelastic1006/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jul '22

07 Jul '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1006 Address: 208.80.154.138 State: CRITICAL Date/Time: Thu Jul 7 16:52:34 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 292 threshold =0.2 breach: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, active_primary_shards: 727, active_shards: 1165, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 292, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 79.95881949210707

1 0

Airflow alert: <TaskInstance: drop_old_data_daily.drop_subgraph_mapping_and_metrics_partitions 2022-07-06T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 07 Jul '22

07 Jul '22

1 0

[airflow] SLA miss on DAG=subgraph_mapping_weekly
by noreply＠wikimedia.org 06 Jul '22

06 Jul '22

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts July 2022