August 2019 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - stat1007/Check the last execution of search-drop-query-clicks is CRITICAL **
by nagios＠icinga1001.wikimedia.org 20 Aug '19

20 Aug '19

Notification Type: PROBLEM Service: Check the last execution of search-drop-query-clicks Host: stat1007 Address: 10.64.21.118 State: CRITICAL Date/Time: Tue Aug 20 00:45:47 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_time… Acknowledged by : Additional Info: connect to address 10.64.21.118 port 5666: Connection refused

1 0

** RECOVERY alert - search.svc.eqiad.wmnet/ElasticSearch unassigned shard check - 9243 is OK **
by nagios＠icinga1001.wikimedia.org 18 Aug '19

18 Aug '19

Notification Type: RECOVERY Service: ElasticSearch unassigned shard check - 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: OK Date/Time: Sun Aug 18 10:25:22 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - All good

1 0

** PROBLEM alert - search.svc.eqiad.wmnet/ElasticSearch unassigned shard check - 9243 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 18 Aug '19

18 Aug '19

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9243 Host: search.svc.eqiad.wmnet Address: 10.2.2.30 State: CRITICAL Date/Time: Sun Aug 18 07:36:27 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - eswiki_content_1521891951[6](2019-08-15T03:43:12.536Z), enwiki_content_1546970425[2](2019-08-15T03:43:02.394Z)

1 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **
by nagios＠icinga1001.wikimedia.org 17 Aug '19

17 Aug '19

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Sat Aug 17 00:29:07 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠icinga1001.wikimedia.org 17 Aug '19

17 Aug '19

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sat Aug 17 00:24:19 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **
by nagios＠icinga1001.wikimedia.org 16 Aug '19

16 Aug '19

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Fri Aug 16 12:29:58 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠icinga1001.wikimedia.org 16 Aug '19

16 Aug '19

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Fri Aug 16 12:22:00 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **
by nagios＠icinga1001.wikimedia.org 16 Aug '19

16 Aug '19

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Thu Aug 15 22:27:30 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠icinga1001.wikimedia.org 16 Aug '19

16 Aug '19

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Thu Aug 15 22:22:42 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?orgId=… Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** RECOVERY alert - cloudelastic1002/ElasticSearch health check for shards on 9200 is OK **
by nagios＠icinga1001.wikimedia.org 16 Aug '19

16 Aug '19

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: OK Date/Time: Thu Aug 15 21:19:26 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: unassigned_shards: 0, number_of_pending_tasks: 0, active_shards: 2195, cluster_name: cloudelastic-chi-eqiad, initializing_shards: 4, active_primary_shards: 733, relocating_shards: 0, number_of_nodes: 4, timed_out: False, active_shards_percent_as_number: 99.8180991359709, number_of_data_nodes: 4, status: yellow, number_of_in_flight_fetch: 0, delayed_unassigned_shards: 0, task_max_waiting_in_queue_millis: 0

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts August 2019