Notification Type: PROBLEM
Service: Mediawiki Cirrussearch update rate - eqiad
Host: icinga1001
Address: 208.80.154.84
State: CRITICAL
Date/Time: Sun May 26 10:20:05 UTC 2019
Notes URLs: https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44…
Additional Info:
CRITICAL: 10.00% of data under the critical threshold [50.0]
Traceback (most recent call last):
File "/srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions", line 153, in <module>
hive.drop_partitions(table, partition_specs_to_drop)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 190, in drop_partitions
return self.query(q, use_tempfile=True)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 355, in query
out = self.script(f.name, check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 365, in script
return self._command(['-f', script], check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 372, in _command
return sh(cmd, check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 121, in sh
.format(command_string, p.returncode), stdout, stderr)
RuntimeError: ('Command: hive --service cli --database discovery -f /tmp/tmp-hive-query-UMWeeO.hiveql failed with error code: 1', '', 'Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nlog4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.\n\nLogging initialized using configuration in file:/etc/hive/conf.analytics-hadoop/hive-log4j.properties\nOK\nTime taken: 0.873 seconds\nFAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Got exception: java.io.IOException Failed to move to trash: hdfs://analytics-hadoop/wmf/data/discovery/query_clicks/hourly/year=2019/month=2/day=19/hour=13\n')
Traceback (most recent call last):
File "/srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions", line 153, in <module>
hive.drop_partitions(table, partition_specs_to_drop)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 190, in drop_partitions
return self.query(q, use_tempfile=True)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 355, in query
out = self.script(f.name, check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 365, in script
return self._command(['-f', script], check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 372, in _command
return sh(cmd, check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 121, in sh
.format(command_string, p.returncode), stdout, stderr)
RuntimeError: ('Command: hive --service cli --database discovery -f /tmp/tmp-hive-query-MEd8qu.hiveql failed with error code: 1', '', 'Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nlog4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.\n\nLogging initialized using configuration in file:/etc/hive/conf.analytics-hadoop/hive-log4j.properties\nOK\nTime taken: 0.876 seconds\nFAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Got exception: java.io.IOException Failed to move to trash: hdfs://analytics-hadoop/wmf/data/discovery/query_clicks/hourly/year=2019/month=2/day=19/hour=12\n')
Notification Type: RECOVERY
Service: ElasticSearch unassigned shard check - 9243
Host: search.svc.eqiad.wmnet
Address: 10.2.2.30
State: OK
Date/Time: Sun May 19 21:13:22 UTC 2019
Notes URLs: https://wikitech.wikimedia.org/wiki/Search%23Administration
Additional Info:
OK - All good