The still rather empty Incident Report is at https://wikitech.wikimedia.org/wiki/Incident_documentation/20141125-EventLo… I'll fill it with more information tomorrow.

Backfilling from the logs is already running, and should also finish tomorrow.

Hi, I am sorry to announce yet another EventLogging outage :-( EventLogging's database writer service failed to write events to the database between ~2014-11-25T03:09 and 2014-11-26T00:03. The recent adhoc increase of EventLogging's database throughput capacity to address the EventLogging database writing bottle-neck came with the known issue of not too robust exit synchronization of threads within the EventLogging database writer. This exit synchronization issue got the database writer stuck and caused the outage. A fix for that known issue is sitting in gerrit since Sunday, but due to the many meetings and discussions around recent EventLogging issues, the fix did not yet get reviewed and deployed. The still rather empty Incident Report is at https://wikitech.wikimedia.org/wiki/Incident_documentation/20141125-EventLo… I'll fill it with more information tomorrow. Backfilling from the logs is already running, and should also finish tomorrow. Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics