- Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - elastic2061/ElasticSearch setting check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9200 Host: elastic2061 Address: 10.192.0.143 State: CRITICAL Date/Time: Thu Mar 28 06:32:39 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2047.codfw.wmnet:9500, elastic2052.codfw.wmnet:9500, elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] does not match [elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] for .(cluster

1 0

** PROBLEM alert - elastic2084/ElasticSearch setting check - 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9200 Host: elastic2084 Address: 10.192.48.56 State: CRITICAL Date/Time: Thu Mar 28 06:32:39 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2047.codfw.wmnet:9500, elastic2052.codfw.wmnet:9500, elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] does not match [elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] for .(cluster

1 0

** PROBLEM alert - elastic2083/ElasticSearch setting check - 9600 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9600 Host: elastic2083 Address: 10.192.32.88 State: CRITICAL Date/Time: Thu Mar 28 06:24:33 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2047.codfw.wmnet:9500, elastic2052.codfw.wmnet:9500, elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] does not match [elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] for .(cluster

1 0

** PROBLEM alert - elastic2108/ElasticSearch setting check - 9600 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9600 Host: elastic2108 Address: 10.192.48.90 State: CRITICAL Date/Time: Thu Mar 28 06:24:33 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2047.codfw.wmnet:9500, elastic2052.codfw.wmnet:9500, elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] does not match [elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] for .(cluster

1 0

** PROBLEM alert - elastic2076/ElasticSearch setting check - 9600 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9600 Host: elastic2076 Address: 10.192.0.206 State: CRITICAL Date/Time: Thu Mar 28 06:24:33 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2047.codfw.wmnet:9500, elastic2052.codfw.wmnet:9500, elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] does not match [elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] for .(cluster

1 0

** PROBLEM alert - elastic2075/ElasticSearch setting check - 9600 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9600 Host: elastic2075 Address: 10.192.0.145 State: CRITICAL Date/Time: Thu Mar 28 06:24:33 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2047.codfw.wmnet:9500, elastic2052.codfw.wmnet:9500, elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] does not match [elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] for .(cluster

1 0

** PROBLEM alert - elastic2080/ElasticSearch setting check - 9600 is CRITICAL **
by nagios＠alert1001.wikimedia.org 28 Mar '24

28 Mar '24

Notification Type: PROBLEM Service: ElasticSearch setting check - 9600 Host: elastic2080 Address: 10.192.16.127 State: CRITICAL Date/Time: Thu Mar 28 06:24:33 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2047.codfw.wmnet:9500, elastic2052.codfw.wmnet:9500, elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] does not match [elastic2073.codfw.wmnet:9500, elastic2086.codfw.wmnet:9500, elastic2092.codfw.wmnet:9500, elastic2100.codfw.wmnet:9500, elastic2106.codfw.wmnet:9500] for .(cluster

1 0

[FIRING:1] SystemdUnitFailed search-platform (wdqs1013:9100 wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service ops critical eqiad prometheus)
by sre-observability＠wikimedia.org 28 Mar '24

28 Mar '24

1 0

[FIRING:1] SystemdUnitFailed search-platform (wdqs1013:9100 wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service ops critical eqiad prometheus)
by sre-observability＠wikimedia.org 27 Mar '24

27 Mar '24

1 0

** ACKNOWLEDGEMENT alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9443 is CRITICAL **
by nagios＠alert1001.wikimedia.org 27 Mar '24

27 Mar '24

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Mar 27 22:36:22 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Ryan Kemper: Trouble talking to master(s) following decoms investigating Additional Info: CRITICAL - elasticsearch https://search.svc.codfw.wmnet:9443/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.codfw.wmnet, port=9443): Read timed out. (read timeout=4)

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts