[Engineering] Change to Monolog logging as seen in Kibana (Logstash) that I forgot to announce

[Bryan Davis]

Jamie filed T126438 [0] which reminded me that I didn't get around to
sending out an announcement email about a configuration change I made
[1] to the MediaWiki Monolog logger recently. The TL;DR is that the
"normalized_message" value associated with a given log event is now
created based on the debug log message *before* any PSR3 placeholders
(e.g. the things in curly braces in "Error connecting to {db_server}:
{error}") are expanded. We have a lot of debug log events which
include highly variable information such as page titles or session
identifiers of some sort. The prior method of grouping on the expanded
messages would mask trending error conditions where each event used a
slightly different string to describe the problem.

When you see a message like "{time} {title}" trending in a dashboard
[2] and want to know what is going on, you can click the magnifying
glass on the right side of the trending gadget to filter the dashboard
to only matching messages. The "all events" view below will show the
fully expanded messages (e.g. "3.11 Space_Race", "5.24
Wikidata:List_of_properties/all", ...).

For some dashboards in Kibana, this change may not be as useful as it
is for others. The prior behavior can be roughly approximated by
aggregating "message.raw" instead of "normalized_message.raw". We can
also create more targeted dashboards which show aggregations of the
placeholder elements themselves rather than the expanded messages
which in some cases may be even more useful. I'd be glad to work with
people who need a bit of help designing a more usable view of some
particular dashboard. Find me on irc or probably even better open a
phab task and ping me. We don't have a project in phab explicitly for
Kibana, but #wikimedia-logstash and #monitoring would be a good
project choices.


[0]: https://phabricator.wikimedia.org/T126438
[1]: https://phabricator.wikimedia.org/rOMWCb356eeb1647751e2037345e19cb658172d9676ab
[2]: https://logstash.wikimedia.org/#/dashboard/elasticsearch/mediawiki

Bryan
-- 
Bryan Davis              Wikimedia Foundation    <bd808 at wikimedia.org>
[[m:User:BDavis_(WMF)]]  Sr Software Engineer            Boise, ID USA
irc: bd808                                        v:415.839.6885 x6855