Thanks Christian.
On Nov 18, 2014, at 5:20 PM, Christian Aistleitner christian@quelltextlich.at wrote:
Hi Ryan,
On Tue, Nov 18, 2014 at 11:48:55AM -0800, Ryan Kaldari wrote: It looks like the same problem is happening now. No new events have been written to the log tables on analytics-store for about the past hour and a half.
Yes, there were at least (maybe more. I'll do a check and full write-up once I've gotten some sleep) EventLogging issues for
- 2014-11-18T18:15 -- 2014-11-18T21:05, and
- 2014-11-18T23:19 -- 2014-11-19T00:07.
Recent EventLogging changes caused the process that writes to the database to hit Python's recursion limit and cause issues.
EventLogging got rolled back to the previously deployed version (Thanks ori!), and is doing fine since.
Proper incident report to follow.
Sorry for the inconveniences, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian@quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics