On Thu, Nov 26, 2015 at 10:46 PM, Ori Livneh <ori@wikimedia.org> wrote:
Seems that eventlog1001 has not received any events since 01:30 UTC on Thursday

http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&c=Miscellaneous+eqiad&h=eventlog1001.eqiad.wmnet&jr=&js=&event=hide&ts=0&v=140128.28&m=bytes_in&vl=bytes%2Fsec&ti=Bytes+Received

This is pretty severe; I'd page if it wasn't a US holiday.

Kafka clients on eventlog1001 were in a "Autocommitting consumer offset" death-loop and not receiving any events from the Kafka brokers. I ran eventloggingctl stop / eventloggingctl start and they recovered. Needs to be investigated more thoroughly. Otto, can you follow up?