It looks like the same problem is happening now. No new events have been written to the log tables on analytics-store for about the past hour and a half. And it looks like the slave db stopped replicating about 6 hours ago. Ryan Kaldari On Tue, Nov 18, 2014 at 9:31 AM, Christian Aistleitner < christian(a)quelltextlich.at&gt; wrote: _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

On Nov 18, 2014, at 5:20 PM, Christian Aistleitner &lt;christian(a)quelltextlich.at&gt; wrote: Hi Ryan, Yes, there were at least (maybe more. I'll do a check and full write-up once I've gotten some sleep) EventLogging issues for * 2014-11-18T18:15 -- 2014-11-18T21:05, and * 2014-11-18T23:19 -- 2014-11-19T00:07. Recent EventLogging changes caused the process that writes to the database to hit Python's recursion limit and cause issues. EventLogging got rolled back to the previously deployed version (Thanks ori!), and is doing fine since. Proper incident report to follow. Sorry for the inconveniences, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian(a)quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics