Discovery-alerts March 2025

discovery-alerts@lists.wikimedia.org

4 participants
734 discussions

[FIRING:1] SystemdUnitFailed search-platform (relforge1003:9100 opensearch_1@relforge-eqiad-small-alpha.service\x2copensearch_1@relforge-eqiad.service ops critical eqiad prometheus)
by sre-observability＠wikimedia.org 31 Mar '25

31 Mar '25

1 0

** RECOVERY alert - elastic1068/Elasticsearch HTTPS for production-search-eqiad is OK **
by nagios＠alert1002.wikimedia.org 31 Mar '25

31 Mar '25

Notification Type: RECOVERY Service: Elasticsearch HTTPS for production-search-eqiad Host: elastic1068 Address: 10.64.0.72 State: OK Date/Time: Mon Mar 31 20:49:57 UTC 2025 Notes URLs: https://wikitech.wikimedia.org/wiki/Search Acknowledged by : Additional Info: SSL OK - Certificate elastic1068.eqiad.wmnet valid until 2025-04-16 12:19:00 +0000 (expires in 15 days)

1 0

** RECOVERY alert - elastic1068/Elasticsearch HTTPS for production-search-omega-eqiad is OK **
by nagios＠alert1002.wikimedia.org 31 Mar '25

31 Mar '25

Notification Type: RECOVERY Service: Elasticsearch HTTPS for production-search-omega-eqiad Host: elastic1068 Address: 10.64.0.72 State: OK Date/Time: Mon Mar 31 20:49:25 UTC 2025 Notes URLs: https://wikitech.wikimedia.org/wiki/Search Acknowledged by : Additional Info: SSL OK - Certificate elastic1068.eqiad.wmnet valid until 2025-04-16 12:19:00 +0000 (expires in 15 days)

1 0

31 Mar '25

1 0

[FIRING:3] CirrusConsumerRerenderFetchErrorRate search-platform (cirrus-streaming-updater k8s critical prometheus)
by sre-observability＠wikimedia.org 31 Mar '25

31 Mar '25

1 0

** PROBLEM alert - relforge1003/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 31 Mar '25

31 Mar '25

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Mon Mar 31 15:37:45 UTC 2025 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 39 threshold =0.15 breach: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, active_primary_shards: 53, active_shards: 68, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 39, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 63.55140186915887

1 0

** PROBLEM alert - relforge1008/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 31 Mar '25

31 Mar '25

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: relforge1008 Address: 10.64.32.100 State: CRITICAL Date/Time: Mon Mar 31 15:37:31 UTC 2025 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 39 threshold =0.15 breach: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, active_primary_shards: 53, active_shards: 68, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 39, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 63.55140186915887

1 0

** PROBLEM alert - relforge1009/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 31 Mar '25

31 Mar '25

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: relforge1009 Address: 10.64.48.120 State: CRITICAL Date/Time: Mon Mar 31 15:37:29 UTC 2025 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 39 threshold =0.15 breach: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, active_primary_shards: 53, active_shards: 68, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 39, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 63.55140186915887

1 0

** PROBLEM alert - relforge1004/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1002.wikimedia.org 31 Mar '25

31 Mar '25

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Mon Mar 31 15:37:23 UTC 2025 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 39 threshold =0.15 breach: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 4, number_of_data_nodes: 4, discovered_master: True, active_primary_shards: 53, active_shards: 68, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 39, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 63.55140186915887

1 0

[FIRING:2] CirrusConsumerRerenderFetchErrorRate search-platform (cirrus-streaming-updater k8s consumer-search critical prometheus)
by sre-observability＠wikimedia.org 31 Mar '25

31 Mar '25

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts March 2025