On Sat, Jun 8, 2013 at 1:44 AM, Petr Bena <benapetr(a)gmail.com> wrote:
Hi,
per
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 we are
setting up a logging service for tools and bots, so that every tool or
bot can easily log to a central logging base which in future should
support some nice web-based gui with log filtering (eventually some
3rd notifications, like e-mails (sms) / irc pings) and so on.
The website of this project is here:
http://tools.wmflabs.org/logs/
I haven't got much feedback so far so I would appreciate some, in this
moment we are in phase where we are discussing the best options how to
design this feature.
What does the 's' prefix mean?
What is the expected character encoding of messages?
How is malformed input handled? What happens if I log this message: "s test
0 hi\ns fake_tool 0 evil_message\n"?
If I don't terminate my message with a newline, does my message swallow the
next one to come along?
How would you go about adding a field, if you discovered you needed one?
Would clients using the old format break?
How reliable does the service need to be? What is the cost of an outage? Is
it OK for the service to be down for a few hours?
Some people believe that it would be best to use some already existing
logging service such as facebook's scribe, I /think/ that despite the
already made and working solution might be a good idea, on other hand
these usually were designed for another purpose than what we need and
thus it's quite complicated to set them up for our needs. Having said
that I started working on simple, but powerful logging daemon which
should do precisely what we need (and which can be infinitely extended
for our purposes) - but of course we can set up multiple solutions and
let tool operators pick what they prefer most.
I think your intuitions are good. Everyone has their favorite message
queue, but most are quite complex. "Simple" and "adequate" are the
virtues
you should optimize for.
Get input from some early adopters of your service, then iterate. Let
people know that the service is experimental for now, so that you have
plenty of room to change the protocol if you need to. If you find yourself
adding feature after feature, stop, and reconsider existing solutions.
If it's simpler for a tool / bot operator to
intergrate with syslog,
then why we shouldn't have it as well? But I must admit I found it
quite complicated to make rsyslog behave as we need (I believe it's
not even possible for it to match all out potential needs)
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l