August 2022 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 03 Aug '22

03 Aug '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Wed Aug 3 01:03:24 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is OK **
by nagios＠alert1001.wikimedia.org 02 Aug '22

02 Aug '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Aug 2 18:34:30 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

** ACKNOWLEDGEMENT alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9443 is CRITICAL **
by nagios＠alert1001.wikimedia.org 02 Aug '22

02 Aug '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch numbers of masters eligible - 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Aug 2 17:50:35 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by Ryan Kemper: https://phabricator.wikimedia.org/T310070 Additional Info: CRITICAL - Found 2 eligible masters.

1 0

** ACKNOWLEDGEMENT alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is CRITICAL **
by nagios＠alert1001.wikimedia.org 02 Aug '22

02 Aug '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Aug 2 17:50:35 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by Ryan Kemper: https://phabricator.wikimedia.org/T310070 Additional Info: CRITICAL - Found 2 eligible masters.

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is CRITICAL **
by nagios＠alert1001.wikimedia.org 02 Aug '22

02 Aug '22

Notification Type: PROBLEM Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Aug 2 17:47:18 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: CRITICAL - Found 2 eligible masters.

1 0

[FIRING:1] RdfStreamingUpdaterFlinkProcessingLatencyIsHigh (WDQS_Streaming_Updater rdf-streaming-updater k8s warning codfw prometheus search-platform)
by sre-observability＠wikimedia.org 02 Aug '22

02 Aug '22

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is OK **
by nagios＠alert1001.wikimedia.org 02 Aug '22

02 Aug '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Aug 2 15:47:52 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9243 is CRITICAL **
by nagios＠alert1001.wikimedia.org 02 Aug '22

02 Aug '22

Notification Type: PROBLEM Service: ElasticSearch numbers of masters eligible - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Aug 2 15:22:27 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: CRITICAL - Found 2 eligible masters.

1 0

Airflow alert: <TaskInstance: drop_old_data_daily.drop_subgraph_mapping_and_metrics_partitions 2022-08-01T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 02 Aug '22

02 Aug '22

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9643 is OK **
by nagios＠alert1001.wikimedia.org 01 Aug '22

01 Aug '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Mon Aug 1 19:58:53 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts August 2022