Where is the source code for this logging
script/software?
*-- *
*Tyler Romeo*
Stevens Institute of Technology, Class of 2016
Major in Computer Science
| tylerromeo(a)gmail.com
On Sat, Jun 8, 2013 at 6:36 AM, Petr Bena <benapetr(a)gmail.com> wrote:
Replies bellow
On Sat, Jun 8, 2013 at 11:58 AM, Ori Livneh <ori(a)wikimedia.org> wrote:
On Sat, Jun 8, 2013 at 1:44 AM, Petr Bena
<benapetr(a)gmail.com> wrote:
Hi,
per
https://bugzilla.wikimedia.org/show_bug.cgi?id=48846 we are
setting up a logging service for tools and bots, so that every tool or
bot can easily log to a central logging base which in future should
support some nice web-based gui with log filtering (eventually some
3rd notifications, like e-mails (sms) / irc pings) and so on.
The website of this project is here:
http://tools.wmflabs.org/logs/
I haven't got much feedback so far so I would appreciate some, in this
moment we are in phase where we are discussing the best options how to
design this feature.
What does the 's' prefix mean?
it is shortcut for "store"
you can either type
s ...
or
store ...
it will do same, or you can use "n" (noreply) that will do same as s
or store but will send no reply on success
What is the expected character encoding of
messages?
You got me. I have no idea, either mono is smart enough to find out or
it's uses some standard encoding by default, like utf-8 or unicode? I
will find out. Google doesn't bring clear answer.
How is malformed input handled? What happens if I
log this message: "s
test
0 hi\ns fake_tool 0 evil_message\n"?
malformed (invalid) input gets response ERROR: <explanation>
malformed (malicious) input is something I am still thinking of, I
don't expect many users to intentionally break logs of others, but I
am thinking of implementing optional authentication. If tool requested
to use authentication (it would subscribe / register a folder somehow)
it could for example get a temporary token that would stored somewhere
like /data/project/logs/tokens/tool and which would be readable only
by tool account.
This token would be updated periodically for security reasons and tool
would be required to provide it together with log message in order to
write to its folder.
But this is something I am still designing and I want to keep it
opt-in as it complicate stuff and many users will likely not require
it. Also we are getting far beyond possibilities of rsyslog now
(another reason why using own daemon could be better).
If I don't terminate my message with a
newline, does my message swallow
the
next one to come along?
I forgot to mention newline isn't required in UDP connection. You can
send multiple log lines using newline or just 1 line and it will be
parsed.
If you terminate TCP connection before writing newline (which means -
buffer is finished, store it - to daemon), it will be likely dropped
(you won't get the STORED response either).
How would you go about adding a field, if you
discovered you needed one?
Would clients using the old format break?
What kind of field do you mean?
How reliable does the service need to be? What is
the cost of an outage?
Is
it OK for the service to be down for a few hours
It's not OK for it to be down for few hours, however the clients
should be able to recognize the server is down if they are using tcp
(because they won't be able to connect).
The wrapper script I made now (logeater) should be optimized to
recognize this as well and keep retrying until successful or
something. Outage is always a problem for any kind of logger.
Some people believe that it would be best to use some already existing
logging service such as facebook's scribe, I /think/ that despite the
already made and working solution might be a good idea, on other hand
these usually were designed for another purpose than what we need and
thus it's quite complicated to set them up for our needs. Having said
that I started working on simple, but powerful logging daemon which
should do precisely what we need (and which can be infinitely extended
for our purposes) - but of course we can set up multiple solutions and
let tool operators pick what they prefer most.
I think your intuitions are good. Everyone has their favorite message
queue, but most are quite complex. "Simple" and "adequate" are the
virtues
you should optimize for.
Get input from some early adopters of your service, then iterate. Let
people know that the service is experimental for now, so that you have
plenty of room to change the protocol if you need to. If you find
yourself
adding feature after feature, stop, and
reconsider existing solutions.
If it's simpler for a tool / bot operator to
intergrate with syslog,
then why we shouldn't have it as well? But I must admit I found it
quite complicated to make rsyslog behave as we need (I believe it's
not even possible for it to match all out potential needs)
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org