More from tsuna about how StumbleUpon uses Kafka:
Begin forwarded message:
From: tsuna tsunanet@gmail.com Subject: Re: Scribe Packaging Effort Date: July 27, 2012 1:08:29 AM EDT To: Andrew Otto otto@wikimedia.org
On Thu, Jul 26, 2012 at 9:01 AM, Andrew Otto otto@wikimedia.org wrote:
What do you guys use Kafka for?
Just as a simple message bus between different components. For example when certain events happen on our site, we create a message corresponding to the event and send it to the appropriate Kafka topic, and consumers interested in these sort of messages can get them and handle them however they want.
In your case it looks like you are shipping logs around and you want to have a pipeline where one of the stages is transforming the messages. I think this should be easy to build with Flume, as they have APIs for sources and sinks, so although I've never done it myself, I expect it would be rather straightforward to write your own Agent that transforms messages and insert it in between the producers and the HDFS sink. This sounds like it would be simpler than dealing with Scribe's problems or throwing Storm into the picture.
-- Benoit "tsuna" Sigoure Software Engineer @ www.StumbleUpon.com