October 2020 - Discovery-alerts - lists.wikimedia.org

Fatal Error - Oozie Job discovery-query-clicks-hourly-discovery.query_clicks_hourly-2020-10-8-6-wf
by oozie＠an-coord1001.eqiad.wmnet 08 Oct '20

08 Oct '20

Error details: - Job id: UNSET - Failing action: aggregate - Error code: UNSET - Error message: UNSET - Hue link: https://hue.wikimedia.org/oozie/list_oozie_workflows Please have a look and take necessary action ! Thanks :) -- Oozie

1 0

Fatal Error - Oozie Job discovery-query-clicks-hourly-discovery.query_clicks_hourly-2020-10-8-4-wf
by oozie＠an-coord1001.eqiad.wmnet 08 Oct '20

08 Oct '20

Error details: - Job id: UNSET - Failing action: aggregate - Error code: UNSET - Error message: UNSET - Hue link: https://hue.wikimedia.org/oozie/list_oozie_workflows Please have a look and take necessary action ! Thanks :) -- Oozie

1 0

Fatal Error - Oozie Job discovery-query-clicks-hourly-discovery.query_clicks_hourly-2020-10-8-2-wf
by oozie＠an-coord1001.eqiad.wmnet 08 Oct '20

08 Oct '20

Error details: - Job id: UNSET - Failing action: aggregate - Error code: UNSET - Error message: UNSET - Hue link: https://hue.wikimedia.org/oozie/list_oozie_workflows Please have a look and take necessary action ! Thanks :) -- Oozie

1 0

** RECOVERY alert - graphite1004/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 07 Oct '20

07 Oct '20

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: OK Date/Time: Wed Oct 7 22:09:11 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1004/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Oct '20

07 Oct '20

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1004 Address: 10.64.16.149 State: CRITICAL Date/Time: Wed Oct 7 22:02:21 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/dashboard/db/elasticsearch-percentiles?panelI… Acknowledged by : Additional Info: CRITICAL: 20.00% of data above the critical threshold [1000.0]

1 0

** RECOVERY alert - relforge1001/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 07 Oct '20

07 Oct '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1001 Address: 10.64.4.13 State: OK Date/Time: Wed Oct 7 21:15:57 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: cluster_name: relforge-eqiad, number_of_data_nodes: 2, active_shards_percent_as_number: 100.0, initializing_shards: 0, timed_out: False, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, active_primary_shards: 83, status: green, unassigned_shards: 0, task_max_waiting_in_queue_millis: 0, active_shards: 104, relocating_shards: 0, number_of_in_flight_fetch: 0, number_of_nodes: 2

1 0

** PROBLEM alert - relforge1001/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Oct '20

07 Oct '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1001 Address: 10.64.4.13 State: CRITICAL Date/Time: Wed Oct 7 21:10:53 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

1 0

** RECOVERY alert - relforge1002/ElasticSearch health check for shards on 9200 is OK **
by nagios＠alert1001.wikimedia.org 07 Oct '20

07 Oct '20

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9200 Host: relforge1002 Address: 10.64.37.21 State: OK Date/Time: Wed Oct 7 21:05:37 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status relforge-eqiad: number_of_pending_tasks: 0, unassigned_shards: 0, number_of_nodes: 2, active_shards: 104, relocating_shards: 0, number_of_in_flight_fetch: 0, timed_out: False, cluster_name: relforge-eqiad, status: green, initializing_shards: 0, delayed_unassigned_shards: 0, number_of_data_nodes: 2, active_primary_shards: 83, active_shards_percent_as_number: 100.0, task_max_waiting_in_queue_millis: 0

1 0

** PROBLEM alert - relforge1002/ElasticSearch health check for shards on 9200 is CRITICAL **
by nagios＠alert1001.wikimedia.org 07 Oct '20

07 Oct '20

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9200 Host: relforge1002 Address: 10.64.37.21 State: CRITICAL Date/Time: Wed Oct 7 21:02:13 UTC 2020 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch inactive shards 48 threshold =0.15 breach: delayed_unassigned_shards: 0, active_shards_percent_as_number: 52.0, cluster_name: relforge-eqiad, number_of_nodes: 1, number_of_data_nodes: 1, timed_out: False, active_shards: 52, active_primary_shards: 52, relocating_shards: 0, task_max_waiting_in_queue_millis: 0, number_of_pending_tasks: 0, initializing_shards: 0, status: red, number_of_in_flight_fetch: 0, unassigned_shards: 48

1 0

Airflow alert: <TaskInstance: mjolnir.make_folds-enwiki-dbn-20180215-query_explorer-pruned_mrmr 2020-09-30T00:00:00+00:00 [failed]>
by noreply＠wikimedia.org 07 Oct '20

07 Oct '20

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts October 2020