That's what I said to you the other day when I
looked a bit at Scribe
and read about how it doesn't work with a newer Thrift and the patches
to make it so haven't been merged.
Yeah totally. The more I look at it and
work with it the more I am worried. I mean, for a simple scalable syslogish type
replacement that just works, it is pretty good. Or, at least it was when we decided to
use it at CouchSurfing years ago. I'm sure for log collection into files, it still
works very well. But we might be trying to do fancier things with it, and it might be a
headache to try to bend it to our will. It also looks like it might have been superseded
by some other options.
Faidon, I'll bring this up at my first ops meeting coming up! Not so much to make a
decision on it, but just to make everyone aware that we are evaluating options, and that
we would love any input anyone has. The main issue might be: if not Scribe, then we
might have go with a Java solution (e.g. Flume, Kafka, etc.). Is ops ok with that?
Domas
Danke! I will try to find him on IRC or
send him an email. Thanks!
-Ao
On Jul 26, 2012, at 11:14 AM, Faidon Liambotis <faidon(a)wikimedia.org> wrote:
> On Thu, Jul 26, 2012 at 09:43:48AM -0400, Andrew Otto wrote:
>> Here's his opinion on Scribe:
>>> I personally gave up on Scribe, I'd recommend that you consider
>>> Flume as a better replacement, that is more supported and developed.
>>> Scribe has never been really well written or maintained, it's just
>>> one of the many hacks that Facebook released.
>>
>> In general, it does seem to be pretty given up on. There have been a
>> couple of pull requests merged in the last year, but beyond that there
>> isn't much activity:
>>
https://github.com/facebook/scribe/commits/master
>
That's what I said to you the other day when I
looked a bit at Scribe
and read about how it doesn't work with a newer Thrift and the patches
to make it so haven't been merged.
>
> It seems abandonded, or at least its public releases are. My 2 cents is
> that this is quite dangerous for us: if it just works we'll never get
> new features; if it fails, we're basically on our own to fix it, same as
> udp2log. I thought the whole point was to to move something that has
> active developers working on it :-)
>
>> It would be really interesting to know how (and if?) Facebook still
>> uses Scribe internally. I'm pretty sure they've done a lot more with
>> Hadoop since 2008-2010 when Scribe was being more actively promoted.
>> Maybe they're using Flume instead now? We need a Facebook insider,
>> anyone know one?
>
> Domas works at Facebook and has been on of the first Wikimedia
> (volunteer) techops, being our main DBA among other things for years. He
> does less things nowadays but still hangs around on IRC channels,
> mailing lists, was at the Berlin Hackathon etc. You should catch-up with
> him on IRC; his nickname is "domas" and he doesn't have a predictable
> timezone :)
>
> Regards,
> Faidon