April 2022 - Discovery-alerts - lists.wikimedia.org

selenium-daily-beta-RelatedArticles - Build # 1387 - Failure!
by Release Engineering 26 Apr '22

26 Apr '22

1 1

selenium-daily-beta-CirrusSearch - Build # 1350 - Failure!
by Release Engineering 21 Apr '22

21 Apr '22

1 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **
by nagios＠alert1001.wikimedia.org 20 Apr '22

20 Apr '22

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Wed Apr 20 08:04:35 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 19 Apr '22

19 Apr '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Tue Apr 19 23:50:04 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 19 Apr '22

19 Apr '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Tue Apr 19 23:27:28 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 19 Apr '22

19 Apr '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Tue Apr 19 22:25:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [600.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 19 Apr '22

19 Apr '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Tue Apr 19 22:05:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 30.00% of data above the critical threshold [600.0]

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9243 is OK **
by nagios＠alert1001.wikimedia.org 19 Apr '22

19 Apr '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Tue Apr 19 09:18:10 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-eqiad: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 36, number_of_data_nodes: 36, active_primary_shards: 1355, active_shards: 3657, relocating_shards: 0, initializing_shards: 55, unassigned_shards: 410, delayed_unassigned_shards: 0, number_of_pending_tasks: 714, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 366354, active_shards_percent_as_number: 88.71906841339155

1 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch health check for shards on 9243 is CRITICAL **
by nagios＠alert1001.wikimedia.org 19 Apr '22

19 Apr '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Tue Apr 19 09:16:10 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 1236 threshold =0.2 breach: cluster_name: production-search-eqiad, status: red, timed_out: False, number_of_nodes: 36, number_of_data_nodes: 36, active_primary_shards: 1347, active_shards: 2886, relocating_shards: 0, initializing_shards: 72, unassigned_shards: 1164, delayed_unassigned_shards: 0, number_of_pending_tasks: 1628, number_of_in_flight_fetch: 36, task_max_waiting_in_queue_millis: 247056, active_shards_percent_as_number: 70.01455604075691

1 0

[FIRING:2] BlazegraphJvmQuakeWarnGC wdqs (node /tmp/wdqs_blazegraph_jvmquake_warn_gc ops warning prometheus search-platform)
by sre-observability＠wikimedia.org 13 Apr '22

13 Apr '22

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts April 2022