July 2022 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 Jul '22

23 Jul '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sat Jul 23 06:45:35 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 30.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 23 Jul '22

23 Jul '22

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Sat Jul 23 06:32:59 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 23 Jul '22

23 Jul '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sat Jul 23 06:30:27 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true false cloudelastic-chi-eqiad old 208.80.154.138 cloudelastic1006:9108 elasticsearch cloudelastic1006-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 23 Jul '22

23 Jul '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true false cloudelastic-chi-eqiad old 208.80.154.138 cloudelastic1006:9108 elasticsearch cloudelastic1006-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 23 Jul '22

23 Jul '22

1 0

Airflow alert: <TaskInstance: drop_old_data_daily.drop_subgraph_mapping_and_metrics_partitions 2022-07-22T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 22 Jul '22

22 Jul '22

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is OK **
by nagios＠alert1001.wikimedia.org 22 Jul '22

22 Jul '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Fri Jul 22 16:23:47 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is CRITICAL **
by nagios＠alert1001.wikimedia.org 22 Jul '22

22 Jul '22

Notification Type: PROBLEM Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Fri Jul 22 16:11:15 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: CRITICAL - Found 2 eligible masters.

1 0

Airflow alert: <TaskInstance: drop_old_data_daily.drop_subgraph_mapping_and_metrics_partitions 2022-07-21T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 21 Jul '22

21 Jul '22

1 0

[airflow] SLA miss on DAG=subgraph_query_mapping_daily
by noreply＠wikimedia.org 21 Jul '22

21 Jul '22

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts July 2022