From: tsuna <tsunanet(a)gmail.com>
Subject: Re: Scribe Packaging Effort
Date: July 27, 2012 1:08:29 AM EDT
To: Andrew Otto <otto(a)wikimedia.org>
On Thu, Jul 26, 2012 at 9:01 AM, Andrew Otto <otto(a)wikimedia.org> wrote:
What do you guys use Kafka for?
Just as a simple message bus between different components. For
example when certain events happen on our site, we create a message
corresponding to the event and send it to the appropriate Kafka topic,
and consumers interested in these sort of messages can get them and
handle them however they want.
In your case it looks like you are shipping logs around and you want
to have a pipeline where one of the stages is transforming the
messages. I think this should be easy to build with Flume, as they
have APIs for sources and sinks, so although I've never done it
myself, I expect it would be rather straightforward to write your own
Agent that transforms messages and insert it in between the producers
and the HDFS sink. This sounds like it would be simpler than dealing
with Scribe's problems or throwing Storm into the picture.
--
Benoit "tsuna" Sigoure
Software Engineer @
www.StumbleUpon.com