August 2022 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch numbers of masters eligible - 9643 is CRITICAL **
by nagios＠alert1001.wikimedia.org 17 Aug '22

17 Aug '22

Notification Type: PROBLEM Service: ElasticSearch numbers of masters eligible - 9643 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Wed Aug 17 18:37:50 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: CRITICAL - Found 2 eligible masters.

1 0

** PROBLEM alert - stat1007/Check unit status of search-drop-query-clicks is CRITICAL **
by nagios＠alert1001.wikimedia.org 16 Aug '22

16 Aug '22

Notification Type: PROBLEM Service: Check unit status of search-drop-query-clicks Host: stat1007 Address: 10.64.21.118 State: CRITICAL Date/Time: Wed Aug 17 03:42:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time… Acknowledged by : Additional Info: CRITICAL: Status of the systemd unit search-drop-query-clicks

1 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **
by nagios＠alert1001.wikimedia.org 16 Aug '22

16 Aug '22

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Wed Aug 17 02:51:37 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 16 Aug '22

16 Aug '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Wed Aug 17 02:35:29 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [600.0]

1 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **
by nagios＠alert1001.wikimedia.org 16 Aug '22

16 Aug '22

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Wed Aug 17 01:31:14 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 16 Aug '22

16 Aug '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Wed Aug 17 01:26:34 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 30.00% of data above the critical threshold [600.0]

1 0

Airflow alert: <TaskInstance: search_satisfaction_daily.wait_for_logs 2022-08-16T00:00:00+00:00 [up_for_retry]>
by noreply＠wikimedia.org 16 Aug '22

16 Aug '22

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch numbers of masters eligible - 9643 is OK **
by nagios＠alert1001.wikimedia.org 16 Aug '22

16 Aug '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9643 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Tue Aug 16 21:41:50 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch numbers of masters eligible - 9643 is CRITICAL **
by nagios＠alert1001.wikimedia.org 16 Aug '22

16 Aug '22

Notification Type: PROBLEM Service: ElasticSearch numbers of masters eligible - 9643 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Tue Aug 16 21:12:42 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: CRITICAL - Found 2 eligible masters.

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch numbers of masters eligible - 9443 is OK **
by nagios＠alert1001.wikimedia.org 16 Aug '22

16 Aug '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9443 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Tue Aug 16 19:23:46 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts August 2022