February 2023 - Discovery-alerts - lists.wikimedia.org

Airflow alert: <TaskInstance: transfer_to_es_hourly.wait_for_recommendations 2023-02-21T11:00:00+00:00 [up_for_retry]>

by noreply＠wikimedia.org

1 year, 2 months

1
0
0 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9243 is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Feb 21 14:28:06 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-codfw: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 36, number_of_data_nodes: 36, active_primary_shards: 1664, active_shards: 4030, relocating_shards: 0, initializing_shards: 140, unassigned_shards: 802, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 81.05390185036202

1 year, 2 months

1
0
0 0

** ACKNOWLEDGEMENT alert - an-airflow1005/Checks that the local airflow scheduler for airflow @search is working properly is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: Checks that the local airflow scheduler for airflow @search is working properly Host: an-airflow1005 Address: 10.64.36.11 State: CRITICAL Date/Time: Tue Feb 21 14:27:22 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow Acknowledged by Brian_King: T327970 Additional Info: CRITICAL: /usr/bin/env AIRFLOW_HOME=/srv/airflow-search /usr/lib/airflow/bin/airflow jobs check --job-type SchedulerJob --hostname an-airflow1005.eqiad.wmnet did not succeed

1 year, 2 months

1
0
0 0

** ACKNOWLEDGEMENT alert - an-airflow1005/Checks that the airflow database for airflow search is working properly is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: Checks that the airflow database for airflow search is working properly Host: an-airflow1005 Address: 10.64.36.11 State: CRITICAL Date/Time: Tue Feb 21 14:27:22 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow Acknowledged by Brian_King: T327970 Additional Info: CRITICAL: /usr/bin/env AIRFLOW_HOME=/srv/airflow-search /usr/lib/airflow/bin/airflow db check did not succeed

1 year, 2 months

1
0
0 0

** ACKNOWLEDGEMENT alert - an-airflow1005/Check systemd state is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: Check systemd state Host: an-airflow1005 Address: 10.64.36.11 State: CRITICAL Date/Time: Tue Feb 21 14:27:22 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by Brian_King: T327970 Additional Info: CRITICAL - degraded: The following units failed: wmf_auto_restart_airflow-kerberos@search.service,wmf_auto_restart_airflow-webserver@search.service

1 year, 2 months

1
0
0 0

** ACKNOWLEDGEMENT alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9243 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Feb 21 14:25:50 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Brian_King: T327991 Additional Info: CRITICAL - elasticsearch inactive shards 1083 threshold =0.2 breach: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 36, number_of_data_nodes: 36, active_primary_shards: 1664, active_shards: 3889, relocating_shards: 0, initializing_shards: 142, unassigned_shards: 941, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 78.21802091713596

1 year, 2 months

1
0
0 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9243 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Feb 21 14:24:20 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 1083 threshold =0.2 breach: cluster_name: production-search-codfw, status: yellow, timed_out: False, number_of_nodes: 36, number_of_data_nodes: 36, active_primary_shards: 1664, active_shards: 3889, relocating_shards: 0, initializing_shards: 142, unassigned_shards: 941, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 78.21802091713596

1 year, 2 months

1
0
0 0

[FIRING:1] RdfStreamingUpdaterFlinkJobUnstable (WDQS_Streaming_Updater rdf-streaming-updater k8s critical codfw prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 2 months

1
0
0 0

[FIRING:1] WdqsStreamingUpdaterFlinkJobNotRunning (WDQS_Streaming_Updater rdf-streaming-updater k8s critical codfw prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 2 months

1
0
0 0

[FIRING:1] WcqsStreamingUpdaterFlinkJobNotRunning (WCQS_Streaming_Updater rdf-streaming-updater k8s critical codfw prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 2 months

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts February 2023