March 2019 - Discovery-alerts - lists.wikimedia.org

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9243 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Mar 27 20:10:00 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch inactive shards 607 threshold =0.15 breach: relocating_shards: 0, delayed_unassigned_shards: 0, task_max_waiting_in_queue_millis: 0, active_shards: 2465, number_of_pending_tasks: 0, timed_out: False, number_of_nodes: 30, unassigned_shards: 569, number_of_in_flight_fetch: 0, cluster_name: production-search-codfw, initializing_shards: 38, active_primary_shards: 1028, status: yellow, number_of_data_nodes: 30, active_shards_percent_as_number: 80.24088541666666

3 years, 12 months

1
0
0 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9443 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9443 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Mar 27 20:09:56 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch inactive shards 556 threshold =0.15 breach: relocating_shards: 0, number_of_nodes: 15, active_primary_shards: 1108, number_of_in_flight_fetch: 0, status: yellow, number_of_pending_tasks: 0, unassigned_shards: 546, number_of_data_nodes: 15, initializing_shards: 10, active_shards_percent_as_number: 83.26813120674089, task_max_waiting_in_queue_millis: 0, timed_out: False, cluster_name: production-search-omega-codfw, active_shards: 2767, delayed_unassigned_shards: 0

3 years, 12 months

1
0
0 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for shards on 9643 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for shards on 9643 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Wed Mar 27 20:09:40 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration Additional Info: CRITICAL - elasticsearch inactive shards 626 threshold =0.15 breach: active_shards_percent_as_number: 80.75030750307504, number_of_data_nodes: 15, cluster_name: production-search-psi-codfw, number_of_pending_tasks: 1, task_max_waiting_in_queue_millis: 0, active_shards: 2626, number_of_in_flight_fetch: 0, initializing_shards: 46, unassigned_shards: 580, active_primary_shards: 1084, number_of_nodes: 15, relocating_shards: 0, delayed_unassigned_shards: 0, status: yellow, timed_out: False

3 years, 12 months

1
0
0 0

** ACKNOWLEDGEMENT alert - icinga1001/Mjolnir bulk update failure check - eqiad is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: ACKNOWLEDGEMENT Service: Mjolnir bulk update failure check - eqiad Host: icinga1001 Address: 208.80.154.84 State: CRITICAL Date/Time: Wed Mar 27 14:39:48 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: 5.59e+05 gt 2

3 years, 12 months

1
0
0 0

** PROBLEM alert - icinga1001/Mjolnir bulk update failure check - eqiad is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: Mjolnir bulk update failure check - eqiad Host: icinga1001 Address: 208.80.154.84 State: CRITICAL Date/Time: Wed Mar 27 07:51:24 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: 5.473e+05 gt 2

3 years, 12 months

1
0
0 0

Cron <hdfs@hadoop-coordinator-2> export PYTHONPATH=${PYTHONPATH}:/srv/deployment/analytics/refinery/python && /srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions -d 90 -D discovery -t query_clicks_hourly, query_clicks_daily >> /var/log/refinery/drop-query-clicks.log

by root＠hadoop-coordinator-2.analytics.eqiad.wmflabs

Traceback (most recent call last): File "/srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions", line 97, in <module> table_location = hive.table_location(table) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 245, in table_location self._tables_init() File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 181, in _tables_init for table in self._tables_get(): File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 165, in _tables_get return self.query('SET hive.cli.print.header=false; SHOW TABLES').splitlines() File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 471, in query return self._command(['-e', query], check_return_code) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 486, in _command return sh(cmd, check_return_code) File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 127, in sh .format(command_string, p.returncode), stdout, stderr) RuntimeError: ('Command: hive --service cli --database discovery -e SET hive.cli.print.header=false; SHOW TABLES failed with error code: 1', '', 'Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nlog4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.\n\nLogging initialized using configuration in file:/etc/hive/conf.analytics-hadoop-labs/hive-log4j.properties\nException in thread "main" java.lang.RuntimeException: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop-coordinator-2.analytics.eqiad.wmflabs/172.16.2.237"; destination host is: "hadoop-master-4.analytics.eqiad.wmflabs":8020; \n\tat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:571)\n\tat org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:695)\n\tat org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:634)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.util.RunJar.run(RunJar.java:226)\n\tat org.apache.hadoop.util.RunJar.main(RunJar.java:141)\nCaused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop-coordinator-2.analytics.eqiad.wmflabs/172.16.2.237"; destination host is: "hadoop-master-4.analytics.eqiad.wmflabs":8020; \n\tat org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1508)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1441)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)\n\tat com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:788)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)\n\tat com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)\n\tat org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2168)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262)\n\tat org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1262)\n\tat org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418)\n\tat org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:665)\n\tat org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:606)\n\tat org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547)\n\t... 8 more\nCaused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]\n\tat org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)\n\tat org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)\n\tat org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)\n\tat org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)\n\tat org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1480)\n\t... 28 more\nCaused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]\n\tat com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)\n\tat org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)\n\tat org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)\n\tat org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)\n\tat org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)\n\tat org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)\n\tat org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)\n\t... 31 more\nCaused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)\n\tat sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)\n\tat sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)\n\tat sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)\n\tat sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)\n\tat sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)\n\tat sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)\n\tat com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)\n\t... 40 more\n')

3 years, 12 months

1
0
0 0

** PROBLEM alert - icinga1001/Mjolnir bulk update failure check - codfw is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: Mjolnir bulk update failure check - codfw Host: icinga1001 Address: 208.80.154.84 State: CRITICAL Date/Time: Wed Mar 27 02:46:06 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: 3.247e+06 gt 2

3 years, 12 months

1
0
0 0

** PROBLEM alert - icinga1001/Mjolnir bulk update failure check - codfw is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: Mjolnir bulk update failure check - codfw Host: icinga1001 Address: 208.80.154.84 State: CRITICAL Date/Time: Tue Mar 26 23:04:15 UTC 2019 Notes URLs: https://grafana.wikimedia.org/d/000000591/elasticsearch-mjolnir-bulk-update… Additional Info: 2.406e+06 gt 2

3 years, 12 months

1
0
0 0

** RECOVERY alert - search.svc.codfw.wmnet/ElasticSearch health check for frozen writes - 9243 is OK **

by nagios＠icinga1001.wikimedia.org

Notification Type: RECOVERY Service: ElasticSearch health check for frozen writes - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: OK Date/Time: Tue Mar 26 22:03:05 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Pausing_Indexing Additional Info: OK - elasticsearch / cirrus frozen writes: no freeze

3 years, 12 months

1
0
0 0

** PROBLEM alert - search.svc.codfw.wmnet/ElasticSearch health check for frozen writes - 9243 is CRITICAL **

by nagios＠icinga1001.wikimedia.org

Notification Type: PROBLEM Service: ElasticSearch health check for frozen writes - 9243 Host: search.svc.codfw.wmnet Address: 10.2.1.30 State: CRITICAL Date/Time: Tue Mar 26 22:01:47 UTC 2019 Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Pausing_Indexing Additional Info: CRITICAL - elasticsearch / cirrus frozen writes: 4:04:59.015935

3 years, 12 months

1
0
0 0

2023

2022

2021

2020

2019

2018

2017

2016

Discovery-alerts March 2019