October 2023 - Discovery-alerts - lists.wikimedia.org

Update on build jenkins-wikidata-query-rdf-maven-release-docker-123
by Release Engineering 20 Oct '23

20 Oct '23

The build 123 for wikidata/query/rdf has finished. See https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-d… for details.

1 0

** RECOVERY alert - an-airflow1005/Check systemd state is OK **
by nagios＠alert1001.wikimedia.org 20 Oct '23

20 Oct '23

Notification Type: RECOVERY Service: Check systemd state Host: an-airflow1005 Address: 10.64.36.11 State: OK Date/Time: Fri Oct 20 09:12:18 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: OK - running: The system is fully operational

1 0

** PROBLEM alert - an-airflow1005/Check systemd state is CRITICAL **
by nagios＠alert1001.wikimedia.org 19 Oct '23

19 Oct '23

Notification Type: PROBLEM Service: Check systemd state Host: an-airflow1005 Address: 10.64.36.11 State: CRITICAL Date/Time: Fri Oct 20 00:04:09 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state Acknowledged by : Additional Info: CRITICAL - degraded: The following units failed: prometheus-node-textfile-prometheus-check-certificate-expiry.service

1 0

[FIRING:1] SystemdUnitFailed search-platform (apifeatureusage1001:9100 puppet-agent-timer.service ops warning eqiad prometheus)
by sre-observability＠wikimedia.org 19 Oct '23

19 Oct '23

1 0

[FIRING:1] RdfStreamingUpdaterFlinkJobUnstable search-platform (WDQS_Streaming_Updater rdf-streaming-updater k8s critical eqiad prometheus)
by sre-observability＠wikimedia.org 19 Oct '23

19 Oct '23

1 0

[FIRING:1] SystemdUnitFailed search-platform (wdqs1024:9100 systemd-timedated.service ops warning eqiad prometheus)
by sre-observability＠wikimedia.org 19 Oct '23

19 Oct '23

1 0

** RECOVERY alert - graphite1005/CirrusSearch eqiad 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 19 Oct '23

19 Oct '23

Notification Type: RECOVERY Service: CirrusSearch eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Thu Oct 19 06:29:23 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** RECOVERY alert - graphite1005/CirrusSearch codfw 95th percentile latency is OK **
by nagios＠alert1001.wikimedia.org 19 Oct '23

19 Oct '23

Notification Type: RECOVERY Service: CirrusSearch codfw 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: OK Date/Time: Thu Oct 19 06:28:01 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: OK: Less than 20.00% above the threshold [500.0]

1 0

** PROBLEM alert - graphite1005/CirrusSearch codfw 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 19 Oct '23

19 Oct '23

Notification Type: PROBLEM Service: CirrusSearch codfw 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Thu Oct 19 06:19:23 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 40.00% of data above the critical threshold [1000.0]

1 0

** PROBLEM alert - graphite1005/CirrusSearch eqiad 95th percentile latency is CRITICAL **
by nagios＠alert1001.wikimedia.org 19 Oct '23

19 Oct '23

Notification Type: PROBLEM Service: CirrusSearch eqiad 95th percentile latency Host: graphite1005 Address: 10.64.16.81 State: CRITICAL Date/Time: Thu Oct 19 06:19:17 UTC 2023 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Health/Activity_Monitoring https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?orgId=1… Acknowledged by : Additional Info: CRITICAL: 50.00% of data above the critical threshold [1000.0]

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts October 2023