Hi,
I am sorry to announce yet another EventLogging outage :-(
EventLogging's database writer service failed to write events to the database between ~2014-11-25T03:09 and 2014-11-26T00:03.
The recent adhoc increase of EventLogging's database throughput capacity to address the EventLogging database writing bottle-neck came with the known issue of not too robust exit synchronization of threads within the EventLogging database writer. This exit synchronization issue got the database writer stuck and caused the outage.
A fix for that known issue is sitting in gerrit since Sunday, but due to the many meetings and discussions around recent EventLogging issues, the fix did not yet get reviewed and deployed.
The still rather empty Incident Report is at
https://wikitech.wikimedia.org/wiki/Incident_documentation/20141125-EventLog...
I'll fill it with more information tomorrow.
Backfilling from the logs is already running, and should also finish tomorrow.
Best regards, Christian