On Fri, Jan 30, 2015 at 11:44 AM, Erik Bernhardson ebernhardson@wikimedia.org wrote:
This is great bryan, thanks for moving this all forward. One exciting part of this logging update is that the log methods also take a context array which (i believe) is imported to kibana and searchable via elasticsearch. Has any effort been put into best practices around what should be in these context arrays?
In my experiences at a $job-1 being able to find related logs via these context values was incredibly useful. We implemented both a global logging context which was used for all logs of the same request including a hash that uniquely identifies the request, the username if logged in, and the route(we don't have a direct equiv to route, Title in some cases). Area specific debugging would add their own info. For example if failing to connect to a remote server it would be included in the logging context so we could easily find all errors connecting to the same machine.
The context data provided with a log message will indeed be passed on to Logstash on the WMF cluster and eventually end up as viewable/searchable in the Kibana frontend. We are using Monolog as the PSR-3 implementation library on the cluster [0]. In the current configuration we are adding several things to all log messages using Monolog's "Processor" classes:
* host => wfHostname() * wiki => wfWikiID() * process_id => getmypid() * uid => random 7 digit hex integer that correlates all log events for a given request * url => $_SERVER['REQUEST_URI'] * ip => $_SERVER['REMOTE_ADDR'] * http_method' => $_SERVER['REQUEST_METHOD'] * server => $_SERVER['SERVER_NAME'] * referrer => $_SERVER['HTTP_REFERER']
It would be great to come up with standard names for other bits of information that might commonly be attached to a log message. 'exception' is a pretty easy one to pick for adding an Exception class to a log event. Monolog will automagically expand any Exception in the log event's context to a nice structure similar to our exception-json log event stream. 'method' and 'class' might be reasonable standard keys to use as well.
[0]: http://noc.wikimedia.org/conf/highlight.php?file=logging.php
Bryan