Discovery-alerts February 2024

discovery-alerts@lists.wikimedia.org

4 participants
399 discussions

** PROBLEM alert - cloudelastic1008/ElasticSearch unassigned shard check - 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9400 Host: cloudelastic1008 Address: 208.80.154.134 State: CRITICAL Date/Time: Sat Feb 3 04:46:31 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - fywiktionary_content_1693308635[0](2024-01-30T20:52:14.400Z), sswiki_content_1682324242[0](2024-01-30T20:52:14.402Z), kawiktionary_general_1682045987[0](2024-01-30T20:52:14.402Z), tawikinews_content_1682339054[0](2024-01-30T20:52:14.403Z), hakwiki_content_1681938141[0](2024-01-30T20:52:14.402Z), ilowiki_content_1682003872[0](2024-01-30T20:52:14.403Z), abwiktionary_content_1692132157[0](2024-01-30T20:52:14.401Z), cywikiquote_content_1692581299[0](2024-01-30T20:52:14.401Z), fiwikivoyage_general_1693144275[0](2024-01-30T20:52:14.399Z), angwikisource_general_1692144613[0](2024-01-30T20:52:14.399Z), iawikibooks_general_1681994016[0](2024-01-30T20:52:14.403Z), mniwiki_general_1682113849[0](2024-01-30T20:52:14.399Z), kowikiquote_general_1682063572[0](2024-01-30T20:52:14.401Z), bxrwiki_content_1692405982[0](2024-01-30T20:52:14.403Z), btmwiktionary_general_1692403885[0](2024-01-30T20:52:14.400Z), pswiktionary_content_1682172419[0](2024-01-30T20:52:14.402Z), hiwikimedia_general_1681961248[0](2024-01-30T20:5

3 months

** PROBLEM alert - cloudelastic1007/ElasticSearch unassigned shard check - 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9400 Host: cloudelastic1007 Address: 208.80.154.11 State: CRITICAL Date/Time: Sat Feb 3 04:32:11 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - blkwiki_general_1692325160[0](2024-01-30T20:52:14.400Z), swwiktionary_content_1682336016[0](2024-01-30T20:52:14.399Z), fiwikiquote_general_1693141971[0](2024-01-30T20:52:14.403Z), tswiki_general_1682369644[0](2024-01-30T20:52:14.398Z), bxrwiki_content_1692405982[0](2024-01-30T20:52:14.403Z), sahwikisource_content_1682223239[0](2024-01-30T20:52:14.402Z), zh_min_nanwikiquote_content_1682432826[0](2024-01-30T20:52:14.402Z), gawikibooks_content_1693313037[0](2024-01-30T20:52:14.399Z), map_bmswiki_content_1682087026[0](2024-01-30T20:52:14.399Z), ttwikibooks_content_1682372612[0](2024-01-30T20:52:14.399Z), fiwikibooks_general_1693141213[0](2024-01-30T20:52:14.399Z), ruewiki_content_1682184981[0](2024-01-30T20:52:14.403Z), rmwiki_content_1682184205[0](2024-01-30T20:52:14.402Z), arwikibooks_general_1692181666[0](2024-01-30T20:52:14.402Z), tawikibooks_content_1682339007[0](2024-01-30T20:52:14.403Z), lijwikisource_general_1682077655[0](2024-01-30T20:52:14.403Z), bawikibooks_general_1692285898[0](2024-01-30T

3 months

** PROBLEM alert - cloudelastic1009/ElasticSearch unassigned shard check - 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9400 Host: cloudelastic1009 Address: 208.80.154.80 State: CRITICAL Date/Time: Sat Feb 3 04:19:19 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - quwikibooks_content_1682184136[0](2024-01-30T20:52:14.403Z), napwikisource_content_1682124288[0](2024-01-30T20:52:14.401Z), newiktionary_content_1682126793[0](2024-01-30T20:52:14.403Z), knwikiquote_general_1682050558[0](2024-01-30T20:52:14.401Z), angwikiquote_content_1692144583[0](2024-01-30T20:52:14.402Z), bewikisource_general_1692300925[0](2024-01-30T20:52:14.401Z), dgawiki_general_first[0](2024-01-30T20:52:14.400Z), bewikibooks_general_1692297309[0](2024-01-30T20:52:14.401Z), ttwiktionary_content_1682372674[0](2024-01-30T20:52:14.401Z), kgwiki_general_1682046219[0](2024-01-30T20:52:14.402Z), fywiktionary_content_1693308635[0](2024-01-30T20:52:14.400Z), labtestwiki_content_1682073064[0](2024-01-30T20:52:14.401Z), cswikiversity_content_1692566020[0](2024-01-30T20:52:14.399Z), angwikisource_general_1692144613[0](2024-01-30T20:52:14.399Z), arwikinews_general_1692183507[0](2024-01-30T20:52:14.402Z), zuwiktionary_content_1682468427[0](2024-01-30T20:52:14.402Z), hiwikimedia_general_1681961248[0](2024-

3 months

** PROBLEM alert - cloudelastic1010/ElasticSearch unassigned shard check - 9400 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch unassigned shard check - 9400 Host: cloudelastic1010 Address: 10.64.48.24 State: CRITICAL Date/Time: Sat Feb 3 04:11:57 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - dgawiki_content_first[0](2024-01-30T20:52:14.401Z), bmwikiquote_general_1692325479[0](2024-01-30T20:52:14.402Z), ilowiki_content_1682003872[0](2024-01-30T20:52:14.403Z), nlwikinews_content_1682132908[0](2024-01-30T20:52:14.398Z), bjnwikiquote_content_first[0](2024-01-30T20:52:14.400Z), napwikisource_content_1682124288[0](2024-01-30T20:52:14.401Z), sswiki_content_1682324242[0](2024-01-30T20:52:14.402Z), bpywiki_general_1692375521[0](2024-01-30T20:52:14.399Z), angwikiquote_content_1692144583[0](2024-01-30T20:52:14.402Z), fiwikivoyage_general_1693144275[0](2024-01-30T20:52:14.399Z), chowiki_content_1692531553[0](2024-01-30T20:52:14.401Z), btmwiktionary_general_1692403885[0](2024-01-30T20:52:14.400Z), hewikinews_general_1681947456[0](2024-01-30T20:52:14.403Z), kowikiquote_general_1682063572[0](2024-01-30T20:52:14.401Z), ltwikiquote_general_1682083320[0](2024-01-30T20:52:14.400Z), cywikiquote_general_1692581318[0](2024-01-30T20:52:14.402Z), hiwikimedia_content_1681961237[0](2024-01-30T20:52:14.402Z), e

3 months

** RECOVERY alert - graphite1005/CirrusSearch more_like codfw 95th percentile latency is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: CirrusSearch more_like codfw 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Fri Feb 2 19:46:51 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [1000.0]

3 months

** PROBLEM alert - graphite1005/CirrusSearch more_like codfw 95th percentile latency is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: CirrusSearch more_like codfw 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Fri Feb 2 19:43:41 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [1500.0]

3 months

** RECOVERY alert - graphite1005/CirrusSearch comp_suggest codfw 95th percentile latency is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: CirrusSearch comp_suggest codfw 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Fri Feb 2 04:50:34 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [100.0]

3 months, 1 week

** PROBLEM alert - graphite1005/CirrusSearch comp_suggest codfw 95th percentile latency is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: CirrusSearch comp_suggest codfw 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Fri Feb 2 04:44:28 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [250.0]

3 months, 1 week

[FIRING:2] SystemdUnitFailed search-platform (elastic2038:9100 ops warning codfw prometheus)

by sre-observability＠wikimedia.org

3 months, 1 week

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts February 2024