Hi,
as users of gerrit & Co. probably noticed, the m2 master database had issues again on 2014-11-22, and caused EventLogging to not be able to write events to the database on 2014-11-22 between about 19:30 and 21:00.
Data for that period has been backfilled from logs again, so the database should again hold good numbers for that period.
Just wanted to let you know what happened, in case you notice drops in graphs / dashboards that got generated between the incident and now.
Sorry for the inconveniences, Christian
Thanks as usual for dealing with this Christian. Things have been very unstable lately.
We're going to have a post mortem on Monday morning of the recent event logging issues and we'll publish results to this list.
-Toby
On Nov 23, 2014, at 3:12 PM, Christian Aistleitner christian@quelltextlich.at wrote:
Hi,
as users of gerrit & Co. probably noticed, the m2 master database had issues again on 2014-11-22, and caused EventLogging to not be able to write events to the database on 2014-11-22 between about 19:30 and 21:00.
Data for that period has been backfilled from logs again, so the database should again hold good numbers for that period.
Just wanted to let you know what happened, in case you notice drops in graphs / dashboards that got generated between the incident and now.
Sorry for the inconveniences, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian@quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
The same seems to be true of labs databases. Wikimetrics has simply been choking on reports, even for regular people (not just our monster recurrent batches).
On Sun, Nov 23, 2014 at 7:00 PM, Toby Negrin tnegrin@wikimedia.org wrote:
Thanks as usual for dealing with this Christian. Things have been very unstable lately.
We're going to have a post mortem on Monday morning of the recent event logging issues and we'll publish results to this list.
-Toby
On Nov 23, 2014, at 3:12 PM, Christian Aistleitner <
christian@quelltextlich.at> wrote:
Hi,
as users of gerrit & Co. probably noticed, the m2 master database had issues again on 2014-11-22, and caused EventLogging to not be able to write events to the database on 2014-11-22 between about 19:30 and 21:00.
Data for that period has been backfilled from logs again, so the database should again hold good numbers for that period.
Just wanted to let you know what happened, in case you notice drops in graphs / dashboards that got generated between the incident and now.
Sorry for the inconveniences, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Kefermarkterstrasze 6a/3 Email: christian@quelltextlich.at 4293 Gutau, Austria Phone: +43 7946 / 20 5 81 Fax: +43 7946 / 20 5 81 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics