October 2022 - Discovery-alerts - lists.wikimedia.org

[airflow] SLA miss on DAG=subgraph_query_mapping_daily
by noreply＠wikimedia.org 21 Oct '22

21 Oct '22

1 0

[airflow] SLA miss on DAG=subgraph_query_metrics_daily
by noreply＠wikimedia.org 21 Oct '22

21 Oct '22

1 0

** RECOVERY alert - relforge1003/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 20 Oct '22

20 Oct '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: relforge1003 Address: 10.64.5.37 State: OK Date/Time: Thu Oct 20 21:25:50 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad-small-alpha: cluster_name: relforge-eqiad-small-alpha, status: green, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 5, active_shards: 10, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

[FIRING:2] CirrusSearchJVMGCYoungPoolInsufficient elasticsearch (true true true false elasticsearch young ops warning prometheus search-platform)
by sre-observability＠wikimedia.org 20 Oct '22

20 Oct '22

1 0

** RECOVERY alert - relforge1004/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 20 Oct '22

20 Oct '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: OK Date/Time: Thu Oct 20 21:11:59 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad-small-alpha: cluster_name: relforge-eqiad-small-alpha, status: green, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 5, active_shards: 10, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - relforge1004/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 20 Oct '22

20 Oct '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: OK Date/Time: Thu Oct 20 21:09:31 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: cluster_name: relforge-eqiad, status: green, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 153, active_shards: 306, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Oct '22

20 Oct '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Thu Oct 20 20:58:10 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Oct '22

20 Oct '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Thu Oct 20 20:57:58 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - relforge1004/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 20 Oct '22

20 Oct '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: OK Date/Time: Thu Oct 20 20:52:28 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad-small-alpha: cluster_name: relforge-eqiad-small-alpha, status: green, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 5, active_shards: 10, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Oct '22

20 Oct '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Thu Oct 20 20:01:33 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 5 threshold =0.15 breach: cluster_name: relforge-eqiad-small-alpha, status: yellow, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 5, active_shards: 5, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 5, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 50.0

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts October 2022