June 2024 - Discovery-alerts - lists.wikimedia.org

[airflow] SLA miss on DAG=export_queries_to_relforge
by search 26 Jun '24

26 Jun '24

1 0

[FIRING:1] CirrusSearchNodeIndexingNotIncreasing elasticsearch search-platform (true true true false production-search-eqiad 10.64.152.2 elastic1105:9108 elasticsearch elastic1105-production-search-eqiad ops warning eqiad prometheus)
by sre-observability＠wikimedia.org 26 Jun '24

26 Jun '24

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch unassigned shard check - 9243 is OK **
by nagios＠alert1001.wikimedia.org 26 Jun '24

26 Jun '24

Notification Type: RECOVERY Service: ElasticSearch unassigned shard check - 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Wed Jun 26 17:32:08 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - All good

1 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch unassigned shard check - 9243 is CRITICAL **
by nagios＠alert1001.wikimedia.org 26 Jun '24

26 Jun '24

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Wed Jun 26 05:32:07 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - dewiki_content_1717167405[2](2024-06-22T22:42:07.829Z)

1 0

Airflow alert: <TaskInstance: query_clicks_hourly.wait_for_cirrus_requests scheduled__2024-06-25T20:00:00+00:00 [failed]>
by search 26 Jun '24

26 Jun '24

1 0

** RECOVERY alert - graphite1005/CirrusSearch comp_suggest eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 24 Jun '24

24 Jun '24

Notification Type: RECOVERY Service: CirrusSearch comp_suggest eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Mon Jun 24 20:24:18 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [100.0]

1 0

** RECOVERY alert - graphite1005/CirrusSearch full_text eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 24 Jun '24

24 Jun '24

Notification Type: RECOVERY Service: CirrusSearch full_text eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Mon Jun 24 20:16:24 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** RECOVERY alert - graphite1005/CirrusSearch more_like eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 24 Jun '24

24 Jun '24

Notification Type: RECOVERY Service: CirrusSearch more_like eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Mon Jun 24 20:15:12 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [1000.0]

1 0

** PROBLEM alert - graphite1005/CirrusSearch full_text eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 24 Jun '24

24 Jun '24

Notification Type: PROBLEM Service: CirrusSearch full_text eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Mon Jun 24 20:08:22 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [1000.0]

1 0

** PROBLEM alert - graphite1005/CirrusSearch comp_suggest eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 24 Jun '24

24 Jun '24

Notification Type: PROBLEM Service: CirrusSearch comp_suggest eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Mon Jun 24 20:08:14 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [250.0]

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts June 2024