On Sat, Jun 8, 2013 at 1:44 AM, Petr Bena benapetr@gmail.com wrote:
Hi,
per https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 we are setting up a logging service for tools and bots, so that every tool or bot can easily log to a central logging base which in future should support some nice web-based gui with log filtering (eventually some 3rd notifications, like e-mails (sms) / irc pings) and so on.
The website of this project is here: http://tools.wmflabs.org/logs/
I haven't got much feedback so far so I would appreciate some, in this moment we are in phase where we are discussing the best options how to design this feature.
What does the 's' prefix mean?
What is the expected character encoding of messages?
How is malformed input handled? What happens if I log this message: "s test 0 hi\ns fake_tool 0 evil_message\n"?
If I don't terminate my message with a newline, does my message swallow the next one to come along?
How would you go about adding a field, if you discovered you needed one? Would clients using the old format break?
How reliable does the service need to be? What is the cost of an outage? Is it OK for the service to be down for a few hours?
Some people believe that it would be best to use some already existing logging service such as facebook's scribe, I /think/ that despite the already made and working solution might be a good idea, on other hand these usually were designed for another purpose than what we need and thus it's quite complicated to set them up for our needs. Having said that I started working on simple, but powerful logging daemon which should do precisely what we need (and which can be infinitely extended for our purposes) - but of course we can set up multiple solutions and let tool operators pick what they prefer most.
I think your intuitions are good. Everyone has their favorite message queue, but most are quite complex. "Simple" and "adequate" are the virtues you should optimize for.
Get input from some early adopters of your service, then iterate. Let people know that the service is experimental for now, so that you have plenty of room to change the protocol if you need to. If you find yourself adding feature after feature, stop, and reconsider existing solutions.
If it's simpler for a tool / bot operator to intergrate with syslog, then why we shouldn't have it as well? But I must admit I found it quite complicated to make rsyslog behave as we need (I believe it's not even possible for it to match all out potential needs)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l