It seems as of November 29th, something in our logging infrastructure has stopped.

Does anyone know what happened?

Debugging:

Looking at the graphs[1] reveals all data points have them plummeted straight down that day and haven't shown any activity since.

Except on December 4th we observed that the unrealistic value of "0" stopped begin echoed as well. After that, the line is absent entirely.

Example:

http://graphite.wikimedia.org/render/?title=addOnloadHook&width=900&height=200&target=aliasByNode(smartSummarize(mw.js.deprecate.addOnloadHook.rate%2C%221d%22%2C%22avg%22)%2C-1)&from=-4weeks

Raw count:

http://graphite.wikimedia.org/render/?width=900&height=300&target=mw.js.deprecate.addOnloadHook.count&from=-4weeks

The software stack:

* graphite:statsd:

- https://github.com/wikimedia/operations-puppet/commits/3a1134921/modules/txstatsd

- https://github.com/wikimedia/operations-puppet/commits/3a1134921/modules/webperf

* EventLogging: https://meta.wikimedia.org/wiki/Schema:DeprecatedUsage

* WikimediaEvents: https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/34e42e1d73/modules/ext.wikimediaEvents.deprecate.js

* mw.log.deprecate: https://github.com/wikimedia/mediawiki/blob/c6131c5df4/resources/src/mediawiki/mediawiki.js#L564-L573

Nothing there stood out to me.

When connecting to tcp://vanadium.eqiad.wmnet:8600 via zmq from tin.eqiad.wmnet directly[2], there's still a steady flow of incoming data via EventLogging. So it must've stagnated somewhere further down the line (on servers I don't have access to).

Best,

— Timo

[1] mw.js.deprecate graphs:

last 5 months: http://codepen.io/Krinkle/full/zyodJ/

last 3 weeks: http://codepen.io/Krinkle/full/cBGCl/

[2] https://gist.github.com/Krinkle/764cbbadbe272c7ca1e0