Discovery-alerts January 2024

discovery-alerts@lists.wikimedia.org

4 participants
269 discussions

[FIRING:1] RdfStreamingUpdaterHighConsumerUpdateLag wdqs search-platform (wdqs1019:9101 jmx_query_service_streaming_updater ops warning eqiad thanos)

by sre-observability＠wikimedia.org

4 months

[FIRING:1] CirrusSearchJobQueueLagTooHigh search-platform (k8s cirrusSearchOtherIndex-mediawiki-job-cirrusSearchOtherIndex warning eqiad prometheus)

by sre-observability＠wikimedia.org

4 months

** RECOVERY alert - logstash2033/OpenSearch health check for shards on 9200 is OK **

by nagios＠alert1001.wikimedia.org

Notification Type: RECOVERY Service: OpenSearch health check for shards on 9200 Host: logstash2033 Address: 10.192.0.16 State: OK Date/Time: Wed Jan 3 20:27:03 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: OK - elasticsearch status production-elk7-codfw: cluster_name: production-elk7-codfw, status: green, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 12, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 677, active_shards: 1561, relocating_shards: 6, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0

4 months

** PROBLEM alert - logstash2033/OpenSearch health check for shards on 9200 is CRITICAL **

by nagios＠alert1001.wikimedia.org

Notification Type: PROBLEM Service: OpenSearch health check for shards on 9200 Host: logstash2033 Address: 10.192.0.16 State: CRITICAL Date/Time: Wed Jan 3 05:18:49 UTC 2024 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Acknowledged by : Additional Info: CRITICAL - elasticsearch http://localhost:9200/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9200): Read timed out. (read timeout=4)

4 months, 1 week