June 2024 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - graphite1005/CirrusSearch more_like eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 24 Jun '24

24 Jun '24

Notification Type: PROBLEM Service: CirrusSearch more_like eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Mon Jun 24 20:07:10 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 30.00% of data above the critical threshold [1500.0]

1 0

[airflow] SLA miss on DAG=image_suggestions_weekly
by search 23 Jun '24

23 Jun '24

1 0

** RECOVERY alert - graphite1005/CirrusSearch more_like eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: CirrusSearch more_like eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Sat Jun 22 22:53:34 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [1000.0]

1 0

** RECOVERY alert - graphite1005/CirrusSearch comp_suggest eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: CirrusSearch comp_suggest eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Sat Jun 22 22:51:40 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [100.0]

1 0

** RECOVERY alert - graphite1005/CirrusSearch full_text eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: CirrusSearch full_text eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Sat Jun 22 22:51:02 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

[FIRING:1] SystemdUnitFailed search-platform (wdqs1018:9100 systemd-timedated.service ops critical eqiad prometheus)
by sre-observability＠wikimedia.org 23 Jun '24

23 Jun '24

1 0

** RECOVERY alert - an-airflow1005/Checks that the local airflow scheduler for airflow @search is working properly is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: Checks that the local airflow scheduler for airflow @search is working properly Host: an-airflow1005 Address: 10.64.36.11 State: OK Date/Time: Sat Jun 22 22:42:48 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow Acknowledged by : Additional Info: OK: /usr/bin/env PYTHONPATH=/srv/deployment/airflow-dags/search AIRFLOW_HOME=/srv/airflow-search /usr/lib/airflow/bin/airflow jobs check --job-type SchedulerJob --hostname an-airflow1005.eqiad.wmnet succeeded

1 0

** RECOVERY alert - cloudelastic1005/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1005 Address: 10.64.0.97 State: OK Date/Time: Sat Jun 22 22:39:10 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, active_primary_shards: 811, active_shards: 1509, relocating_shards: 2, initializing_shards: 2, unassigned_shards: 110, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 93.09068476249229

1 0

** PROBLEM alert - cloudelastic1005/ElasticSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 Jun '24

23 Jun '24

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1005 Address: 10.64.0.97 State: CRITICAL Date/Time: Sat Jun 22 22:38:18 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4)

1 0

[FIRING:1] RdfStreamingUpdaterFlinkProcessingLatencyIsHigh search-platform (WDQS_Streaming_Updater rdf-streaming-updater k8s wikidata warning eqiad prometheus)
by sre-observability＠wikimedia.org 23 Jun '24

23 Jun '24

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts June 2024