Discovery-alerts March 2019

discovery-alerts@lists.wikimedia.org

4 participants
58 discussions

Cron <hdfs@hadoop-coordinator-2> export PYTHONPATH=${PYTHONPATH}:/srv/deployment/analytics/refinery/python && /srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions -d 90 -D discovery -t query_clicks_hourly, query_clicks_daily >> /var/log/refinery/drop-query-clicks.log
by root＠hadoop-coordinator-2.analytics.eqiad.wmflabs 28 Mar '19

28 Mar '19

Traceback (most recent call last): File "/srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions", line 97, in <module> table_location = hive.table_location(table) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 245, in table_location self._tables_init() File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 181, in _tables_init for table in self._tables_get(): File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 165, in _tables_get return self.query('SET hive.cli.print.header=false; SHOW TABLES').splitlines() File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 471, in query return self._command(['-e', query], check_return_code) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 486, in _command return sh(cmd, check_return_code) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 127, in sh .format(command_string, p.returncode), stdout, stderr) RuntimeError: ('Command: hive --service cli --database discovery -e SET hive.cli.print.header=false; SHOW TABLES failed with error code: 1', '', 'Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nlog4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.\n\nLogging initialized using configuration in file:/etc/hive/conf.analytics-hadoop-labs/hive-log4j.properties\nException in thread "main" java.lang.RuntimeException: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop-coordinator-2.analytics.eqiad.wmflabs/172.16.2.237"; destination host is: "hadoop-master-4.analytics.eqiad.wmflabs":8020; \n\tat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:571)\n\tat org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:695)\n\tat org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:634)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.util.RunJar.run(RunJar.java:226)\n\tat org.apache.hadoop.util.RunJar.main(RunJar.java:141)\nCaused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop-coordinator-2.analytics.eqiad.wmflabs/172.16.2.237"; destination host is: "hadoop-master-4.analytics.eqiad.wmflabs":8020; \n\tat org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1508)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1441)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)\n\tat com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:788)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)\n\tat com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)\n\tat org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2168)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262)\n\tat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1262)\n\tat org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418)\n\tat org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:665)\n\tat org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:606)\n\tat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547)\n\t... 8 more\nCaused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]\n\tat org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)\n\tat org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)\n\tat org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)\n\tat org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)\n\tat org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1480)\n\t... 28 more\nCaused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]\n\tat com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)\n\tat org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)\n\tat org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)\n\tat org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)\n\tat org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)\n\tat org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)\n\tat org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)\n\t... 31 more\nCaused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)\n\tat sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)\n\tat sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)\n\tat sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)\n\tat sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)\n\tat sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)\n\tat sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)\n\tat com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)\n\t... 40 more\n')

1 0

** RECOVERY alert - icinga1001/Mjolnir bulk update failure check - eqiad is OK **
by nagios＠icinga1001.wikimedia.org 28 Mar '19

28 Mar '19

Notification Type: RECOVERY Service: Mjolnir bulk update failure check - eqiad Host: icinga1001 Address: 208.80.154.84 State: OK Date/Time: Thu Mar 28 07:51:17 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: (C)2 gt (W)1 gt 0

1 0

** RECOVERY alert - icinga1001/Mjolnir bulk update failure check - codfw is OK **
by nagios＠icinga1001.wikimedia.org 28 Mar '19

28 Mar '19

Notification Type: RECOVERY Service: Mjolnir bulk update failure check - codfw Host: icinga1001 Address: 208.80.154.84 State: OK Date/Time: Thu Mar 28 04:36:33 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: (C)2 gt (W)1 gt 0

1 0

27 Mar '19

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is OK **
by nagios＠icinga1001.wikimedia.org 27 Mar '19

27 Mar '19

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Wed Mar 27 20:33:54 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - elasticsearch status production-search-psi-codfw: active_shards: 2775, relocating_shards: 0, number_of_nodes: 15, status: yellow, timed_out: False, cluster_name: production-search-psi-codfw, number_of_in_flight_fetch: 0, delayed_unassigned_shards: 4, number_of_pending_tasks: 0, active_shards_percent_as_number: 85.33210332103322, number_of_data_nodes: 15, active_primary_shards: 1084, unassigned_shards: 471, task_max_waiting_in_queue_millis: 0, initializing_shards: 6

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9243 is OK **
by nagios＠icinga1001.wikimedia.org 27 Mar '19

27 Mar '19

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Wed Mar 27 20:30:26 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - elasticsearch status production-search-codfw: task_max_waiting_in_queue_millis: 0, relocating_shards: 0, initializing_shards: 31, unassigned_shards: 429, active_shards_percent_as_number: 85.02604166666666, number_of_in_flight_fetch: 0, number_of_nodes: 30, number_of_pending_tasks: 0, number_of_data_nodes: 30, cluster_name: production-search-codfw, delayed_unassigned_shards: 0, status: yellow, timed_out: False, active_shards: 2612, active_primary_shards: 1028

1 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9443 is OK **
by nagios＠icinga1001.wikimedia.org 27 Mar '19

27 Mar '19

Notification Type: RECOVERY Service: ElasticSearch health check for shards on 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Wed Mar 27 20:27:46 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: OK - elasticsearch status production-search-omega-codfw: timed_out: False, number_of_nodes: 15, initializing_shards: 6, relocating_shards: 0, number_of_data_nodes: 15, delayed_unassigned_shards: 0, task_max_waiting_in_queue_millis: 0, number_of_pending_tasks: 0, active_primary_shards: 1108, active_shards: 2838, number_of_in_flight_fetch: 0, cluster_name: production-search-omega-codfw, active_shards_percent_as_number: 85.40475473969305, status: yellow, unassigned_shards: 479

1 0

** ACKNOWLEDGEMENT alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9443 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Mar '19

27 Mar '19

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Mar 27 20:12:42 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch inactive shards 556 threshold =0.15 breach: delayed_unassigned_shards: 0, number_of_data_nodes: 15, task_max_waiting_in_queue_millis: 0, initializing_shards: 10, timed_out: False, unassigned_shards: 546, number_of_nodes: 15, active_shards: 2767, number_of_pending_tasks: 0, active_shards_percent_as_number: 83.26813120674089, active_primary_shards: 1108, number_of_in_flight_fetch: 0, relocating_shards: 0, cluster_name: production-search-omega-codfw, status: yellow

1 0

** ACKNOWLEDGEMENT alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Mar '19

27 Mar '19

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Mar 27 20:12:42 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch inactive shards 620 threshold =0.15 breach: unassigned_shards: 575, delayed_unassigned_shards: 0, number_of_nodes: 15, active_shards_percent_as_number: 80.93480934809348, status: yellow, active_shards: 2632, timed_out: False, number_of_data_nodes: 15, relocating_shards: 0, cluster_name: production-search-psi-codfw, task_max_waiting_in_queue_millis: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, active_primary_shards: 1084, initializing_shards: 45

1 0

** ACKNOWLEDGEMENT alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9243 is CRITICAL **
by nagios＠icinga1001.wikimedia.org 27 Mar '19

27 Mar '19

Notification Type: ACKNOWLEDGEMENT Service: ElasticSearch health check for shards on 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Mar 27 20:12:42 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch inactive shards 594 threshold =0.15 breach: status: yellow, cluster_name: production-search-codfw, active_shards_percent_as_number: 80.6640625, unassigned_shards: 556, number_of_pending_tasks: 0, relocating_shards: 0, timed_out: False, number_of_in_flight_fetch: 0, number_of_data_nodes: 30, number_of_nodes: 30, active_primary_shards: 1028, delayed_unassigned_shards: 0, initializing_shards: 38, active_shards: 2478, task_max_waiting_in_queue_millis: 0

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts March 2019