Hi, TL;DR: If you think your Hive queries are currently taking longer than usual, please find qchris in IRC, and if he is not responsive, kindly ask someone with root on stat1002 (like Ops) to kill the process java -Dproc_balancer -Xmx1000m [...] ----------------------------------------------------- Data in the Analytics cluster is not evenly distributed. Some data nodes are >90% full, while others are half empty. Data nodes that are >90% full are considered unhealthy and no longer contribute to the pool of available resources. So unhealty data nodes no longer contribute to the total available memory in the cluster. There are other motivations too, but the latter item alone is enough motivation to keep the data nodes balanced and hence healthy. Rebalancing is running since 2015-02-26, but situation is getting worse quicker than rebalancing can rebalance. We've been up to 5 unhealthy nodes. Since we're missing their memory, I decided that we should rebalance more aggressively. Hence, I bumped the rebalancer's capacity, and nodes are recovering and getting healthy again. I am monitoring the increased-capacity rebalancer closely, but in case you're getting blocked by it without me noticing, please find me in IRC and let me know, so I can turn the rebalancer's capacity down. Or if you find me unresponsive, please find someone with root on stat1002 (like Ops) and ask thon to kill the process java -Dproc_balancer -Xmx1000m [...] on stat1002. Have fun, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------