Discovery-alerts March 2019

discovery-alerts@lists.wikimedia.org

4 participants
58 discussions

** RECOVERY alert - icinga2001/Cirrus Update lag check - codfw is OK **

by nagios＠icinga2001.wikimedia.org

Notification Type: RECOVERY Service: Cirrus Update lag check - codfw Host: icinga2001 Address: 208.80.153.74 State: OK Date/Time: Fri Mar 15 10:01:25 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?orgId=1&from=n… Additional Info: (C)2 gt (W)1 gt 0

5 years, 1 month

** RECOVERY alert - icinga2001/Mjolnir bulk update failure check - eqiad is OK **

by nagios＠icinga2001.wikimedia.org

Notification Type: RECOVERY Service: Mjolnir bulk update failure check - eqiad Host: icinga2001 Address: 208.80.153.74 State: OK Date/Time: Fri Mar 15 09:58:45 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: (C)2 gt (W)1 gt 0

5 years, 1 month

** RECOVERY alert - icinga2001/Mjolnir bulk update failure check - codfw is OK **

by nagios＠icinga2001.wikimedia.org

Notification Type: RECOVERY Service: Mjolnir bulk update failure check - codfw Host: icinga2001 Address: 208.80.153.74 State: OK Date/Time: Fri Mar 15 09:58:45 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: (C)2 gt (W)1 gt 0

5 years, 1 month

Cron <hdfs@hadoop-coordinator-2> export PYTHONPATH=${PYTHONPATH}:/srv/deployment/analytics/refinery/python && /srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions -d 90 -D discovery -t query_clicks_hourly, query_clicks_daily >> /var/log/refinery/drop-query-clicks.log

by root＠hadoop-coordinator-2.analytics.eqiad.wmflabs

Traceback (most recent call last): File "/srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions", line 97, in <module> table_location = hive.table_location(table) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 245, in table_location self._tables_init() File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 181, in _tables_init for table in self._tables_get(): File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 165, in _tables_get return self.query('SET hive.cli.print.header=false; SHOW TABLES').splitlines() File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 471, in query return self._command(['-e', query], check_return_code) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 486, in _command return sh(cmd, check_return_code) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 127, in sh .format(command_string, p.returncode), stdout, stderr) RuntimeError: ('Command: hive --service cli --database discovery -e SET hive.cli.print.header=false; SHOW TABLES failed with error code: 1', '', 'Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nlog4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.\n\nLogging initialized using configuration in file:/etc/hive/conf.analytics-hadoop-labs/hive-log4j.properties\nException in thread "main" java.lang.RuntimeException: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop-coordinator-2.analytics.eqiad.wmflabs/172.16.2.237"; destination host is: "hadoop-master-4.analytics.eqiad.wmflabs":8020; \n\tat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:571)\n\tat org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:695)\n\tat org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:634)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.util.RunJar.run(RunJar.java:226)\n\tat org.apache.hadoop.util.RunJar.main(RunJar.java:141)\nCaused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop-coordinator-2.analytics.eqiad.wmflabs/172.16.2.237"; destination host is: "hadoop-master-4.analytics.eqiad.wmflabs":8020; \n\tat org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1508)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1441)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)\n\tat com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:788)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)\n\tat com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)\n\tat org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2168)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262)\n\tat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1262)\n\tat org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418)\n\tat org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:665)\n\tat org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:606)\n\tat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547)\n\t... 8 more\nCaused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]\n\tat org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)\n\tat org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)\n\tat org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)\n\tat org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)\n\tat org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1480)\n\t... 28 more\nCaused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]\n\tat com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)\n\tat org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)\n\tat org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)\n\tat org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)\n\tat org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)\n\tat org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)\n\tat org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)\n\t... 31 more\nCaused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)\n\tat sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)\n\tat sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)\n\tat sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)\n\tat sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)\n\tat sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)\n\tat sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)\n\tat com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)\n\t... 40 more\n')

5 years, 1 month

** PROBLEM alert - icinga2001/Cirrus Update lag check - eqiad is CRITICAL **

by nagios＠icinga2001.wikimedia.org

Notification Type: PROBLEM Service: Cirrus Update lag check - eqiad Host: icinga2001 Address: 208.80.153.74 State: CRITICAL Date/Time: Thu Mar 14 18:04:53 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?orgId=1&from=n… Additional Info: (null)

5 years, 1 month

** PROBLEM alert - icinga2001/Cirrus Update lag check - codfw is CRITICAL **

by nagios＠icinga2001.wikimedia.org

Notification Type: PROBLEM Service: Cirrus Update lag check - codfw Host: icinga2001 Address: 208.80.153.74 State: CRITICAL Date/Time: Thu Mar 14 18:03:07 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000484/kafka-consumer-lag?orgId=1&from=n… Additional Info: (null)

5 years, 1 month

** PROBLEM alert - icinga2001/Mjolnir bulk update failure check - codfw is CRITICAL **

by nagios＠icinga2001.wikimedia.org

Notification Type: PROBLEM Service: Mjolnir bulk update failure check - codfw Host: icinga2001 Address: 208.80.153.74 State: CRITICAL Date/Time: Thu Mar 14 17:58:11 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: (null)

5 years, 1 month

** PROBLEM alert - icinga2001/Mjolnir bulk update failure check - eqiad is CRITICAL **

by nagios＠icinga2001.wikimedia.org

Notification Type: PROBLEM Service: Mjolnir bulk update failure check - eqiad Host: icinga2001 Address: 208.80.153.74 State: CRITICAL Date/Time: Thu Mar 14 17:58:11 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: (null)

5 years, 1 month

2024

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts March 2019