Notification Type: PROBLEM
Service: Check the last execution of refinery-drop-query-clicks
Host: an-coord1001
Address: 10.64.21.104
State: CRITICAL
Date/Time: Tue May 28 03:32:45 UTC 2019
Notes URLs:
Additional Info:
CRITICAL: Status of the systemd unit refinery-drop-query-clicks
Traceback (most recent call last):
File "/srv/deployment/analytics/refinery/bin/refinery-drop-hive-partitions", line 153, in <module>
hive.drop_partitions(table, partition_specs_to_drop)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 190, in drop_partitions
return self.query(q, use_tempfile=True)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 355, in query
out = self.script(f.name, check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 365, in script
return self._command(['-f', script], check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/hive.py", line 372, in _command
return sh(cmd, check_return_code)
File "/srv/deployment/analytics/refinery/python/refinery/util.py", line 121, in sh
.format(command_string, p.returncode), stdout, stderr)
RuntimeError: ('Command: hive --service cli --database discovery -f /tmp/tmp-hive-query-wrWZ0m.hiveql failed with error code: 1', '', 'Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nPicked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8\nlog4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.\n\nLogging initialized using configuration in file:/etc/hive/conf.analytics-hadoop/hive-log4j.properties\nOK\nTime taken: 0.894 seconds\nFAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Got exception: java.io.IOException Failed to move to trash: hdfs://analytics-hadoop/wmf/data/discovery/query_clicks/hourly/year=2019/month=2/day=19/hour=14\n')
Notification Type: RECOVERY
Service: Mediawiki Cirrussearch update rate - eqiad
Host: icinga1001
Address: 208.80.154.84
State: OK
Date/Time: Sun May 26 10:41:09 UTC 2019
Notes URLs: https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44…
Additional Info:
OK: Less than 1.00% under the threshold [80.0]
Notification Type: RECOVERY
Service: Mediawiki Cirrussearch update rate - codfw
Host: icinga1001
Address: 208.80.154.84
State: OK
Date/Time: Sun May 26 10:39:41 UTC 2019
Notes URLs: https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44…
Additional Info:
OK: Less than 1.00% under the threshold [80.0]
Notification Type: PROBLEM
Service: Mediawiki Cirrussearch update rate - codfw
Host: icinga1001
Address: 208.80.154.84
State: CRITICAL
Date/Time: Sun May 26 10:21:23 UTC 2019
Notes URLs: https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?panelId=44…
Additional Info:
CRITICAL: 30.00% of data under the critical threshold [50.0]