June 2024 - Discovery-alerts - lists.wikimedia.org

PROBLEM Host elastic2088 - PING CRITICAL - Packet loss = 100%
by nagios＠alert1001.wikimedia.org 20 Jun '24

20 Jun '24

Notification Type: PROBLEM Host: elastic2088 State: DOWN Address: 10.192.0.91 Info: PING CRITICAL - Packet loss = 100% Date/Time: Thu Jun 20 18:04:03 UTC 2024 Acknowledged by :

1 0

RECOVERY Host elastic2088 - PING WARNING - Packet loss = 90%, RTA = 33.83 ms
by nagios＠alert1001.wikimedia.org 20 Jun '24

20 Jun '24

Notification Type: RECOVERY Host: elastic2088 State: UP Address: 10.192.0.91 Info: PING WARNING - Packet loss = 90%, RTA = 33.83 ms Date/Time: Thu Jun 20 17:44:53 UTC 2024 Acknowledged by :

1 0

PROBLEM Host elastic2088 - PING CRITICAL - Packet loss = 100%
by nagios＠alert1001.wikimedia.org 20 Jun '24

20 Jun '24

Notification Type: PROBLEM Host: elastic2088 State: DOWN Address: 10.192.0.91 Info: PING CRITICAL - Packet loss = 100% Date/Time: Thu Jun 20 17:40:37 UTC 2024 Acknowledged by :

1 0

RECOVERY Host elastic2088 - PING WARNING - Packet loss = 60%, RTA = 0.41 ms
by nagios＠alert1001.wikimedia.org 20 Jun '24

20 Jun '24

Notification Type: RECOVERY Host: elastic2088 State: UP Address: 10.192.0.91 Info: PING WARNING - Packet loss = 60%, RTA = 0.41 ms Date/Time: Thu Jun 20 17:37:47 UTC 2024 Acknowledged by :

1 0

** RECOVERY alert - relforge1004/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 20 Jun '24

20 Jun '24

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: OK Date/Time: Thu Jun 20 16:03:42 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 1016, active_shards: 1754, relocating_shards: 0, initializing_shards: 2, unassigned_shards: 244, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 87.7

1 0

** PROBLEM alert - relforge1004/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Jun '24

20 Jun '24

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1004 Address: 10.64.21.126 State: CRITICAL Date/Time: Thu Jun 20 16:02:42 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 728 threshold =0.15 breach: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 1016, active_shards: 1272, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 728, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 63.6

1 0

** RECOVERY alert - relforge1003/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 20 Jun '24

20 Jun '24

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: OK Date/Time: Thu Jun 20 15:58:41 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 1016, active_shards: 1841, relocating_shards: 0, initializing_shards: 1, unassigned_shards: 158, delayed_unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.05

1 0

** PROBLEM alert - relforge1003/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 20 Jun '24

20 Jun '24

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1003 Address: 10.64.5.37 State: CRITICAL Date/Time: Thu Jun 20 15:55:41 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 650 threshold =0.15 breach: cluster_name: relforge-eqiad, status: yellow, timed_out: False, number_of_nodes: 2, number_of_data_nodes: 2, active_primary_shards: 1016, active_shards: 1350, relocating_shards: 0, initializing_shards: 2, unassigned_shards: 648, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 67.5

1 0

[FIRING:2] SystemdUnitFailed search-platform (elastic2099:9100 ops critical codfw prometheus)
by sre-observability＠wikimedia.org 20 Jun '24

20 Jun '24

1 0

[FIRING:1] CirrusSearchNodeIndexingNotIncreasing elasticsearch search-platform (true true true false production-search-codfw 10.192.32.218 elastic2099:9108 elasticsearch elastic2099-production-search-codfw ops warning codfw prometheus)
by sre-observability＠wikimedia.org 20 Jun '24

20 Jun '24

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts June 2024