On Fri, Jan 30, 2015 at 11:44 AM, Erik Bernhardson
<ebernhardson(a)wikimedia.org> wrote:
This is great bryan, thanks for moving this all
forward. One exciting part
of this logging update is that the log methods also take a context array
which (i believe) is imported to kibana and searchable via elasticsearch.
Has any effort been put into best practices around what should be in these
context arrays?
In my experiences at a $job-1 being able to find related logs via these
context values was incredibly useful. We implemented both a global logging
context which was used for all logs of the same request including a hash
that uniquely identifies the request, the username if logged in, and the
route(we don't have a direct equiv to route, Title in some cases). Area
specific debugging would add their own info. For example if failing to
connect to a remote server it would be included in the logging context so
we could easily find all errors connecting to the same machine.
The context data provided with a log message will indeed be passed on
to Logstash on the WMF cluster and eventually end up as
viewable/searchable in the Kibana frontend. We are using Monolog as
the PSR-3 implementation library on the cluster [0]. In the current
configuration we are adding several things to all log messages using
Monolog's "Processor" classes:
* host => wfHostname()
* wiki => wfWikiID()
* process_id => getmypid()
* uid => random 7 digit hex integer that correlates all log events for
a given request
* url => $_SERVER['REQUEST_URI']
* ip => $_SERVER['REMOTE_ADDR']
* http_method' => $_SERVER['REQUEST_METHOD']
* server => $_SERVER['SERVER_NAME']
* referrer => $_SERVER['HTTP_REFERER']
It would be great to come up with standard names for other bits of
information that might commonly be attached to a log message.
'exception' is a pretty easy one to pick for adding an Exception class
to a log event. Monolog will automagically expand any Exception in the
log event's context to a nice structure similar to our exception-json
log event stream. 'method' and 'class' might be reasonable standard
keys to use as well.
[0]:
http://noc.wikimedia.org/conf/highlight.php?file=logging.php
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA
irc: bd808 v:415.839.6885 x6855