October 2022 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - elastic2075/ElasticSearch setting check - 9600 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch setting check - 9600 Host: elastic2075 Address: 10.192.0.145 State: CRITICAL Date/Time: Thu Oct 6 22:13:13 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2025.codfw.wmnet:9300, elastic2031.codfw.wmnet:9300, elastic2042.codfw.wmnet:9300, elastic2074.codfw.wmnet:9300, elastic2081.codfw.wmnet:9300] does not match [elastic2042.codfw.wmnet:9300, elastic2061.codfw.wmnet:9300, elastic2074.codfw.wmnet:9300, elastic2081.codfw.wmnet:9300, elastic2084.codfw.wmnet:9300] for .(cluster

1 year, 7 months

1
0
0 0

** PROBLEM alert - elastic1073/ElasticSearch setting check - 9600 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch setting check - 9600 Host: elastic1073 Address: 10.64.0.78 State: CRITICAL Date/Time: Thu Oct 6 22:08:45 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic1054.eqiad.wmnet:9300, elastic1074.eqiad.wmnet:9300, elastic1081.eqiad.wmnet:9300] does not match [elastic1054.eqiad.wmnet:9300, elastic1074.eqiad.wmnet:9300, elastic1081.eqiad.wmnet:9300, elastic1094.eqiad.wmnet:9300, elastic1100.eqiad.wmnet:9300] for .(cluster

1 year, 7 months

1
0
0 0

** PROBLEM alert - elastic2080/ElasticSearch setting check - 9600 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch setting check - 9600 Host: elastic2080 Address: 10.192.16.127 State: CRITICAL Date/Time: Thu Oct 6 22:08:45 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2025.codfw.wmnet:9300, elastic2031.codfw.wmnet:9300, elastic2042.codfw.wmnet:9300, elastic2074.codfw.wmnet:9300, elastic2081.codfw.wmnet:9300] does not match [elastic2042.codfw.wmnet:9300, elastic2061.codfw.wmnet:9300, elastic2074.codfw.wmnet:9300, elastic2081.codfw.wmnet:9300, elastic2084.codfw.wmnet:9300] for .(cluster

1 year, 7 months

1
0
0 0

[FIRING:1] CirrusSearchJVMGCYoungPoolInsufficient elasticsearch (true true true false production-search-omega-eqiad 10.64.130.7 elastic1089:9110 elasticsearch elastic1089-production-search-omega-eqiad young ops warning eqiad prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 7 months

1
0
0 0

** RECOVERY alert - graphite1004/Number of backend failures per minute from CirrusSearch is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Thu Oct 6 15:28:41 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [300.0]

1 year, 7 months

1
0
0 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Thu Oct 6 15:03:58 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 year, 7 months

1
0
0 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Thu Oct 6 14:59:12 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 30.00% of data above the critical threshold [1000.0]

1 year, 7 months

1
0
0 0

** PROBLEM alert - graphite1004/Number of backend failures per minute from CirrusSearch is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: Number of backend failures per minute from CirrusSearch Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Thu Oct 6 14:57:22 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [600.0]

1 year, 7 months

1
0
0 0

[FIRING:2] CirrusSearchJVMGCYoungPoolInsufficient elasticsearch (true true true elasticsearch young ops warning eqiad prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 7 months

1
0
0 0

[FIRING:1] MjolnirUpdateFailureRateExceedesThreshold (ops warning eqiad prometheus search-platform)

by sre-observability＠wikimedia.org

1 year, 7 months

1
0
0 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts October 2022