Where is the source code for this logging script/software?
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
On Sat, Jun 8, 2013 at 6:36 AM, Petr Bena benapetr@gmail.com wrote:
Replies bellow
On Sat, Jun 8, 2013 at 11:58 AM, Ori Livneh ori@wikimedia.org wrote:
On Sat, Jun 8, 2013 at 1:44 AM, Petr Bena benapetr@gmail.com wrote:
Hi,
per https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 we are setting up a logging service for tools and bots, so that every tool or bot can easily log to a central logging base which in future should support some nice web-based gui with log filtering (eventually some 3rd notifications, like e-mails (sms) / irc pings) and so on.
The website of this project is here: http://tools.wmflabs.org/logs/
I haven't got much feedback so far so I would appreciate some, in this moment we are in phase where we are discussing the best options how to design this feature.
What does the 's' prefix mean?
it is shortcut for "store"
you can either type
s ... or store ...
it will do same, or you can use "n" (noreply) that will do same as s or store but will send no reply on success
What is the expected character encoding of messages?
You got me. I have no idea, either mono is smart enough to find out or it's uses some standard encoding by default, like utf-8 or unicode? I will find out. Google doesn't bring clear answer.
How is malformed input handled? What happens if I log this message: "s
test
0 hi\ns fake_tool 0 evil_message\n"?
malformed (invalid) input gets response ERROR: <explanation>
malformed (malicious) input is something I am still thinking of, I don't expect many users to intentionally break logs of others, but I am thinking of implementing optional authentication. If tool requested to use authentication (it would subscribe / register a folder somehow) it could for example get a temporary token that would stored somewhere like /data/project/logs/tokens/tool and which would be readable only by tool account.
This token would be updated periodically for security reasons and tool would be required to provide it together with log message in order to write to its folder.
But this is something I am still designing and I want to keep it opt-in as it complicate stuff and many users will likely not require it. Also we are getting far beyond possibilities of rsyslog now (another reason why using own daemon could be better).
If I don't terminate my message with a newline, does my message swallow
the
next one to come along?
I forgot to mention newline isn't required in UDP connection. You can send multiple log lines using newline or just 1 line and it will be parsed.
If you terminate TCP connection before writing newline (which means - buffer is finished, store it - to daemon), it will be likely dropped (you won't get the STORED response either).
How would you go about adding a field, if you discovered you needed one? Would clients using the old format break?
What kind of field do you mean?
How reliable does the service need to be? What is the cost of an outage?
Is
it OK for the service to be down for a few hours
It's not OK for it to be down for few hours, however the clients should be able to recognize the server is down if they are using tcp (because they won't be able to connect).
The wrapper script I made now (logeater) should be optimized to recognize this as well and keep retrying until successful or something. Outage is always a problem for any kind of logger.
Some people believe that it would be best to use some already existing logging service such as facebook's scribe, I /think/ that despite the already made and working solution might be a good idea, on other hand these usually were designed for another purpose than what we need and thus it's quite complicated to set them up for our needs. Having said that I started working on simple, but powerful logging daemon which should do precisely what we need (and which can be infinitely extended for our purposes) - but of course we can set up multiple solutions and let tool operators pick what they prefer most.
I think your intuitions are good. Everyone has their favorite message queue, but most are quite complex. "Simple" and "adequate" are the
virtues
you should optimize for.
Get input from some early adopters of your service, then iterate. Let people know that the service is experimental for now, so that you have plenty of room to change the protocol if you need to. If you find
yourself
adding feature after feature, stop, and reconsider existing solutions.
If it's simpler for a tool / bot operator to intergrate with syslog, then why we shouldn't have it as well? But I must admit I found it quite complicated to make rsyslog behave as we need (I believe it's not even possible for it to match all out potential needs)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l