Hi,
in the week from 2014-09-08–2014-09-14, Andrew and Jeff worked on the following items around the Analytics Cluster and Analytics related Ops:
* Logstash logs from Analytics Cluster * More investigation around analytics1021 partition leader drop-outs * Feasibility check on upgrading stat1002 to trusty
(details below)
Have fun, Christian
* Logstash logs from Analytics Cluster
Logging via gelf got enabled again and is now puppetized. Also names of threads in log messages now get normalized, which makes it way easier to filter.
* More investigation around analytics1021 partition leader drop-outs
Logs from recent analytics1021 drop-outs have been analyzed, but no clear culprit has been identified yet.
* Feasibility check on upgrading stat1002 to trusty
After the stat1003 upgrade to trusty a few weeks back, users asked to upgrade stat1002 to trusty too. However, stat1002 runs Hadoop clients, and Cloudera does not provide Hadoop packages for trusty yet, so upgrading is not too straight forward. Currently, the best way forward seems to be a dist-upgrade, but leaving Hadoop client packages at precise. This approach worked on a labs test instance, but that would put stat1002 in version limbo between precise and trusty. Once another pair of Ops-eyes looked over the approach and agreed to it, stat1002 can get upgraded.