August 2022 - Discovery-alerts - lists.wikimedia.org

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is OK **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Wed Aug 3 18:21:39 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is CRITICAL **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: PROBLEM Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Aug 3 17:26:17 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: CRITICAL - Found 2 eligible masters.

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9643 is OK **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Wed Aug 3 17:14:47 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9643 is CRITICAL **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: PROBLEM Service: ElasticSearch numbers of masters eligible - 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Aug 3 15:16:25 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: CRITICAL - Found 2 eligible masters.

1 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Wed Aug 3 05:36:26 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Wed Aug 3 05:19:56 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Wed Aug 3 04:14:12 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 30.00% of data above the critical threshold [600.0]

1 0

selenium-daily-beta-CirrusSearch - Build # 1453 - Failure!
by Release Engineering 03 Aug '22

03 Aug '22

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Wed Aug 3 02:15:38 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

Airflow alert: <TaskInstance: drop_old_data_daily.drop_subgraph_mapping_and_metrics_partitions 2022-08-02T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 03 Aug '22

03 Aug '22

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts August 2022