March 2024 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - elastic2086/ElasticSearch setting check - 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9400 Host: elastic2086 Address: 10.192.48.179 State: CRITICAL Date/Time: Thu Mar 28 18:32:39 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2025.codfw.wmnet:9300, elastic2031.codfw.wmnet:9300, elastic2042.codfw.wmnet:9300] does not match [elastic2061.codfw.wmnet:9300, elastic2074.codfw.wmnet:9300, elastic2081.codfw.wmnet:9300, elastic2084.codfw.wmnet:9300, elastic2093.codfw.wmnet:9300] for .(cluster

1 0

** PROBLEM alert - elastic2092/ElasticSearch setting check - 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9400 Host: elastic2092 Address: 10.192.16.228 State: CRITICAL Date/Time: Thu Mar 28 18:32:39 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2025.codfw.wmnet:9300, elastic2031.codfw.wmnet:9300, elastic2042.codfw.wmnet:9300] does not match [elastic2061.codfw.wmnet:9300, elastic2074.codfw.wmnet:9300, elastic2081.codfw.wmnet:9300, elastic2084.codfw.wmnet:9300, elastic2093.codfw.wmnet:9300] for .(cluster

1 0

[FIRING:2] RdfStreamingUpdaterHighConsumerUpdateLag wdqs search-platform (wdqs1013:9101 jmx_query_service_streaming_updater thanos-rule warning thanos)
by sre-observability＠wikimedia.org 28 Mar '24

28 Mar '24

1 0

** PROBLEM alert - logstash1012/OpenSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: logstash1012 Address: 10.64.48.177 State: CRITICAL Date/Time: Thu Mar 28 15:51:56 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Max retries exceeded with url: /_cluster/health (Caused by NewConnectionError(urllib3.connection.HTTPConnection object at 0x7fcf3822c280: Failed to establish a new connection: [Errno 111] Connection refused))

1 0

[FIRING:1] SystemdUnitFailed search-platform (wdqs1013:9100 wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service ops critical eqiad prometheus)
by sre-observability＠wikimedia.org 28 Mar '24

28 Mar '24

1 0

[FIRING:1] CirrusSearchJVMGCYoungPoolInsufficient elasticsearch search-platform (true true true false production-search-omega-codfw 10.192.0.97 elastic2090:9110 elasticsearch elastic2090-production-search-omega-codfw young ops warning codfw prometheus)
by sre-observability＠wikimedia.org 28 Mar '24

28 Mar '24

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9443 is OK **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Thu Mar 28 14:58:22 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 25, number_of_data_nodes: 25, active_primary_shards: 1659, active_shards: 4881, relocating_shards: 0, initializing_shards: 6, unassigned_shards: 89, delayed_unassigned_shards: 0, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 52, active_shards_percent_as_number: 98.09083601286174

1 0

** RECOVERY alert - relforge1004/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: relforge1004 Address: 10.64.21.126 State: OK Date/Time: Thu Mar 28 14:54:32 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad-small-alpha: cluster_name: relforge-eqiad-small-alpha, status: green, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 9, active_shards: 16, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** RECOVERY alert - relforge1003/ElasticSearch health check for shards on 9400 is OK **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9400 Host: relforge1003 Address: 10.64.5.37 State: OK Date/Time: Thu Mar 28 14:54:22 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad-small-alpha: cluster_name: relforge-eqiad-small-alpha, status: green, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 9, active_shards: 16, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9443 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Thu Mar 28 14:53:22 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 4976 threshold =0.2 breach: cluster_name: production-search-omega-codfw, status: red, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 18, active_primary_shards: 0, active_shards: 0, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 4976, delayed_unassigned_shards: 0, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 19689, active_shards_percent_as_number: 0.0

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts March 2024