June 2022 - Discovery-alerts - lists.wikimedia.org

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 13 Jun '22

13 Jun '22

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Mon Jun 13 06:29:15 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.32.29 elastic1080:9108 elasticsearch elastic1080-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 13 Jun '22

13 Jun '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Mon Jun 13 06:17:35 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 70.00% of data above the critical threshold [1000.0]

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

Airflow alert: <TaskInstance: cirrus_namespace_map_daily.fetch_namespace_map 2022-06-12T01:00:00+00:00 [failed]>
by noreply＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 13 Jun '22

13 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 12 Jun '22

12 Jun '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sun Jun 12 08:35:04 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 80.00% of data above the critical threshold [600.0]

1 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **
by nagios＠alert1001.wikimedia.org 12 Jun '22

12 Jun '22

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Sun Jun 12 08:18:50 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 12 Jun '22

12 Jun '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sun Jun 12 07:39:16 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [600.0]

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 12 Jun '22

12 Jun '22

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Sun Jun 12 06:24:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.32.29 elastic1080:9108 elasticsearch elastic1080-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 12 Jun '22

12 Jun '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sun Jun 12 06:11:30 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 60.00% of data above the critical threshold [1000.0]

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

Airflow alert: <TaskInstance: cirrus_namespace_map_daily.fetch_namespace_map 2022-06-11T01:00:00+00:00 [failed]>
by noreply＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1004:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 12 Jun '22

12 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 11 Jun '22

11 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 11 Jun '22

11 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 11 Jun '22

11 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 11 Jun '22

11 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 11 Jun '22

11 Jun '22

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 11 Jun '22

11 Jun '22

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Sat Jun 11 06:27:20 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.32.29 elastic1080:9108 elasticsearch elastic1080-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 11 Jun '22

11 Jun '22

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 11 Jun '22

11 Jun '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sat Jun 11 06:16:18 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 80.00% of data above the critical threshold [1000.0]

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 11 Jun '22

11 Jun '22

1 0

Airflow alert: <TaskInstance: cirrus_namespace_map_daily.fetch_namespace_map 2022-06-10T01:00:00+00:00 [failed]>
by noreply＠wikimedia.org 11 Jun '22

11 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 10 Jun '22

10 Jun '22

1 0

** ACKNOWLEDGEMENT alert - cloudelastic1006/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1006 Address: 208.80.154.138 State: CRITICAL Date/Time: Fri Jun 10 21:46:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by daniel_zahn: https://phabricator.wikimedia.org/T309648 Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

** ACKNOWLEDGEMENT alert - cloudelastic1005/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1005 Address: 208.80.154.9 State: CRITICAL Date/Time: Fri Jun 10 21:46:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by daniel_zahn: https://phabricator.wikimedia.org/T309648 Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

** ACKNOWLEDGEMENT alert - cloudelastic1004/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1004 Address: 208.80.155.107 State: CRITICAL Date/Time: Fri Jun 10 21:46:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by daniel_zahn: https://phabricator.wikimedia.org/T309648 Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

** ACKNOWLEDGEMENT alert - cloudelastic1003/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1003 Address: 208.80.154.83 State: CRITICAL Date/Time: Fri Jun 10 21:46:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by daniel_zahn: https://phabricator.wikimedia.org/T309648 Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

** ACKNOWLEDGEMENT alert - cloudelastic1001/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: CRITICAL Date/Time: Fri Jun 10 21:46:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by daniel_zahn: https://phabricator.wikimedia.org/T309648 Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

** ACKNOWLEDGEMENT alert - cloudelastic1002/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: CRITICAL Date/Time: Fri Jun 10 21:46:40 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by daniel_zahn: https://phabricator.wikimedia.org/T309648 Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 10 Jun '22

10 Jun '22

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Fri Jun 10 14:57:01 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 100.00% of data above the critical threshold [600.0]

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 10 Jun '22

10 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 10 Jun '22

10 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 10 Jun '22

10 Jun '22

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Fri Jun 10 06:17:47 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.32.29 elastic1080:9108 elasticsearch elastic1080-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 10 Jun '22

10 Jun '22

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 10 Jun '22

10 Jun '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Fri Jun 10 06:09:11 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 60.00% of data above the critical threshold [1000.0]

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 10 Jun '22

10 Jun '22

1 0

Airflow alert: <TaskInstance: cirrus_namespace_map_daily.fetch_namespace_map 2022-06-09T01:00:00+00:00 [failed]>
by noreply＠wikimedia.org 10 Jun '22

10 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 09 Jun '22

09 Jun '22

1 0

** RECOVERY alert - relforge1003/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 09 Jun '22

09 Jun '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: OK Date/Time: Thu Jun 9 19:24:28 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: cluster_name: relforge-eqiad, status: green, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 140, active_shards: 280, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - relforge1003/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 09 Jun '22

09 Jun '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: relforge1003 Address: 10.64.5.37 State: OK Date/Time: Thu Jun 9 19:24:22 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad-small-alpha: cluster_name: relforge-eqiad-small-alpha, status: green, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 5, active_shards: 10, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** PROBLEM alert - relforge1003/ElasticSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 09 Jun '22

09 Jun '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Thu Jun 9 19:04:52 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 5 threshold =0.15 breach: cluster_name: relforge-eqiad-small-alpha, status: yellow, timed_out: False, number_of_nodes: 1, number_of_data_nodes: 1, active_primary_shards: 5, active_shards: 5, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 5, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 50.0

1 0

** PROBLEM alert - relforge1003/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 09 Jun '22

09 Jun '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Thu Jun 9 19:02:50 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 137 threshold =0.15 breach: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 1, number_of_data_nodes: 1, active_primary_shards: 140, active_shards: 140, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 137, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 50.54151624548736

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 09 Jun '22

09 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 09 Jun '22

09 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 09 Jun '22

09 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 09 Jun '22

09 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.32.29 elastic1080:9108 elasticsearch elastic1080-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 09 Jun '22

09 Jun '22

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 09 Jun '22

09 Jun '22

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Thu Jun 9 06:10:59 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 09 Jun '22

09 Jun '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Thu Jun 9 06:08:39 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 09 Jun '22

09 Jun '22

1 0

Airflow alert: <TaskInstance: cirrus_namespace_map_daily.fetch_namespace_map 2022-06-08T01:00:00+00:00 [failed]>
by noreply＠wikimedia.org 09 Jun '22

09 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

Airflow alert: <TaskInstance: transfer_to_es_hourly.wait_for_recommendations 2022-06-08 13:00:00+00:00 [up_for_retry]>
by noreply＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

Airflow alert: <TaskInstance: transfer_to_es_hourly.wait_for_ores_predictions 2022-06-08T13:00:00+00:00 [up_for_retry]>
by noreply＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

Airflow alert: <TaskInstance: cirrus_namespace_map_daily.fetch_namespace_map 2022-06-07T01:00:00+00:00 [failed]>
by noreply＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 08 Jun '22

08 Jun '22

1 0

** RECOVERY alert - cloudelastic1002/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1002 Address: 208.80.154.133 State: OK Date/Time: Tue Jun 7 20:26:04 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-omega-eqiad: number_of_nodes: 6, number_of_data_nodes: 6, unassigned_shards: 0, cluster_name: cloudelastic-omega-eqiad, active_primary_shards: 768, number_of_in_flight_fetch: 0, timed_out: False, relocating_shards: 2, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0, active_shards: 1538, initializing_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, status: green

1 0

** PROBLEM alert - cloudelastic1002/ElasticSearch health check for shards on 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9400 Host: cloudelastic1002 Address: 208.80.154.133 State: CRITICAL Date/Time: Tue Jun 7 20:23:46 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Max retries exceeded with url: /_cluster/health (Caused by NewConnectionError(requests.packages.urllib3.connection.HTTPConnection object at 0x7fc8d48cd630: Failed to establish a new connection: [Errno 111] Connection refused,))

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

** PROBLEM alert - cloudelastic1004/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1004 Address: 208.80.155.107 State: CRITICAL Date/Time: Tue Jun 7 09:34:32 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

** PROBLEM alert - cloudelastic1003/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1003 Address: 208.80.154.83 State: CRITICAL Date/Time: Tue Jun 7 06:49:17 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.32.29 elastic1080:9108 elasticsearch elastic1080-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Tue Jun 7 06:19:37 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Tue Jun 7 06:17:19 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

** PROBLEM alert - cloudelastic1005/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1005 Address: 208.80.154.9 State: CRITICAL Date/Time: Tue Jun 7 05:54:57 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

** PROBLEM alert - cloudelastic1002/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1002 Address: 208.80.154.133 State: CRITICAL Date/Time: Tue Jun 7 05:49:13 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

** PROBLEM alert - cloudelastic1006/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1006 Address: 208.80.154.138 State: CRITICAL Date/Time: Tue Jun 7 05:38:51 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

** PROBLEM alert - cloudelastic1001/ElasticSearch unassigned shard check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Jun '22

07 Jun '22

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9200 Host: cloudelastic1001 Address: 208.80.154.8 State: CRITICAL Date/Time: Tue Jun 7 05:30:17 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z), commonswiki_file_1647920262[11](2022-05-31T16:52:02.429Z)

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

Airflow alert: <TaskInstance: cirrus_namespace_map_daily.fetch_namespace_map 2022-06-06T01:00:00+00:00 [failed]>
by noreply＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 07 Jun '22

07 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 06 Jun '22

06 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 06 Jun '22

06 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 06 Jun '22

06 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 06 Jun '22

06 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 06 Jun '22

06 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 06 Jun '22

06 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 05 Jun '22

05 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 05 Jun '22

05 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 05 Jun '22

05 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 05 Jun '22

05 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 05 Jun '22

05 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 05 Jun '22

05 Jun '22

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 05 Jun '22

05 Jun '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Sun Jun 5 00:02:29 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 100.00% of data above the critical threshold [600.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 04 Jun '22

04 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 04 Jun '22

04 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 04 Jun '22

04 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 04 Jun '22

04 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 04 Jun '22

04 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 04 Jun '22

04 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 03 Jun '22

03 Jun '22

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 03 Jun '22

03 Jun '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Fri Jun 3 18:00:12 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **
by nagios＠alert1001.wikimedia.org 03 Jun '22

03 Jun '22

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Fri Jun 3 17:55:38 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [600.0]

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 03 Jun '22

03 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 03 Jun '22

03 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 03 Jun '22

03 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 03 Jun '22

03 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 03 Jun '22

03 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 02 Jun '22

02 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 02 Jun '22

02 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true false cloudelastic-chi-eqiad old 208.80.154.138 cloudelastic1006:9108 elasticsearch cloudelastic1006-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 02 Jun '22

02 Jun '22

1 0

[FIRING:2] CirrusSearchHighOldGCFrequency cloudelastic (true true true cloudelastic-chi-eqiad old elasticsearch ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 02 Jun '22

02 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency cloudelastic (true true true true cloudelastic-chi-eqiad old 208.80.155.107 cloudelastic1004:9108 elasticsearch cloudelastic1004-cloudelastic-chi-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 01 Jun '22

01 Jun '22

1 0

** RECOVERY alert - cloudelastic1005/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 01 Jun '22

01 Jun '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1005 Address: 208.80.154.9 State: OK Date/Time: Wed Jun 1 22:08:18 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: task_max_waiting_in_queue_millis: 0, number_of_pending_tasks: 0, initializing_shards: 5, timed_out: False, number_of_in_flight_fetch: 0, status: red, unassigned_shards: 2, cluster_name: cloudelastic-chi-eqiad, active_shards_percent_as_number: 99.5397764628534, active_primary_shards: 758, active_shards: 1514, number_of_nodes: 6, number_of_data_nodes: 6, relocating_shards: 0, delayed_unassigned_shards: 0

1 0

** PROBLEM alert - cloudelastic1005/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 01 Jun '22

01 Jun '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1005 Address: 208.80.154.9 State: CRITICAL Date/Time: Wed Jun 1 22:06:08 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - cloudelastic1005/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 01 Jun '22

01 Jun '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1005 Address: 208.80.154.9 State: OK Date/Time: Wed Jun 1 21:59:08 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status cloudelastic-chi-eqiad: number_of_data_nodes: 6, number_of_in_flight_fetch: 0, number_of_pending_tasks: 1, delayed_unassigned_shards: 0, number_of_nodes: 6, active_shards_percent_as_number: 99.07955292570676, active_primary_shards: 758, active_shards: 1507, initializing_shards: 10, relocating_shards: 0, cluster_name: cloudelastic-chi-eqiad, task_max_waiting_in_queue_millis: 0, timed_out: False, status: red, unassigned_shards: 4

1 0

** PROBLEM alert - cloudelastic1005/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 01 Jun '22

01 Jun '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: cloudelastic1005 Address: 208.80.154.9 State: CRITICAL Date/Time: Wed Jun 1 21:56:58 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

[FIRING:1] BlazegraphFreeAllocatorsDecreasingRapidly wdqs (wdqs1007:9193 blazegraph ops critical eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 01 Jun '22

01 Jun '22

1 0

[FIRING:1] CirrusSearchHighOldGCFrequency elasticsearch (true true true false production-search-eqiad old 10.64.16.50 elastic1076:9108 elasticsearch elastic1076-production-search-eqiad ops warning eqiad prometheus search-platform)
by sre-observability＠wikimedia.org 01 Jun '22

01 Jun '22

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts June 2022