July 2022 - Discovery-alerts - lists.wikimedia.org

Airflow alert: <TaskInstance: drop_old_data_daily.drop_subgraph_mapping_and_metrics_partitions 2022-07-15T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 16 Jul '22

16 Jul '22

1 0

[FIRING:2] RdfStreamingUpdaterFlinkProcessingLatencyIsHigh (WDQS_Streaming_Updater rdf-streaming-updater k8s warning prometheus search-platform)
by sre-observability＠wikimedia.org 15 Jul '22

15 Jul '22

1 0

** RECOVERY alert - elastic2047/ElasticSearch setting check - 9400 is OK **
by nagios＠alert1001.wikimedia.org 15 Jul '22

15 Jul '22

Notification Type: RECOVERY Service: ElasticSearch setting check - 9400 Host: elastic2047 Address: 10.192.32.177 State: OK Date/Time: Fri Jul 15 09:24:38 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - All good!

1 0

Airflow alert: <TaskInstance: drop_old_data_daily.drop_subgraph_mapping_and_metrics_partitions 2022-07-14T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 15 Jul '22

15 Jul '22

1 0

** ACKNOWLEDGEMENT alert - elastic2047/ElasticSearch setting check - 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 14 Jul '22

14 Jul '22

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch setting check - 9400 Host: elastic2047 Address: 10.192.32.177 State: CRITICAL Date/Time: Thu Jul 14 21:45:48 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by Ryan Kemper: https://phabricator.wikimedia.org/T289135 Additional Info: CRITICAL - [elastic2027.codfw.wmnet:9700, elastic2029.codfw.wmnet:9700, elastic2049.codfw.wmnet:9700] does not match [elastic2027.codfw.wmnet:9700, elastic2029.codfw.wmnet:9700, elastic2054.codfw.wmnet:9700] for .(cluster

1 0

** PROBLEM alert - elastic2047/ElasticSearch setting check - 9400 is CRITICAL **
by nagios＠alert1001.wikimedia.org 14 Jul '22

14 Jul '22

Notification Type: PROBLEM Service: ElasticSearch setting check - 9400 Host: elastic2047 Address: 10.192.32.177 State: CRITICAL Date/Time: Thu Jul 14 21:23:11 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - [elastic2027.codfw.wmnet:9700, elastic2029.codfw.wmnet:9700, elastic2049.codfw.wmnet:9700] does not match [elastic2027.codfw.wmnet:9700, elastic2029.codfw.wmnet:9700, elastic2054.codfw.wmnet:9700] for .(cluster

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9643 is OK **
by nagios＠alert1001.wikimedia.org 14 Jul '22

14 Jul '22

Notification Type: RECOVERY Service: ElasticSearch numbers of masters eligible - 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Thu Jul 14 20:44:05 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: OK - All good

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is OK **
by nagios＠alert1001.wikimedia.org 14 Jul '22

14 Jul '22

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Thu Jul 14 20:37:53 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, active_primary_shards: 1534, active_shards: 4026, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 575, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 87.5027168006955

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch numbers of masters eligible - 9643 is CRITICAL **
by nagios＠alert1001.wikimedia.org 14 Jul '22

14 Jul '22

Notification Type: PROBLEM Service: ElasticSearch numbers of masters eligible - 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Thu Jul 14 20:36:33 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Expected_eligible_masters_chec… Acknowledged by : Additional Info: CRITICAL - Found 2 eligible masters.

1 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **
by nagios＠alert1001.wikimedia.org 14 Jul '22

14 Jul '22

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Thu Jul 14 20:25:37 UTC 2022 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch https://search.svc.codfw.wmnet:9643/_cluster/health error while fetching: HTTPSConnectionPool(host=search.svc.codfw.wmnet, port=9643): Read timed out. (read timeout=4)

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts July 2022