I wonder in future if it might be practical useful for test failures like
this to automatically revert changes that made them or at least submit
patches to revert them.... that way it's clear how and when things should
be reverted.
On 6 Mar 2014 18:09, "Chris McMahon" <cmcmahon(a)wikimedia.org> wrote:
On Thu, Mar 6, 2014 at 6:07 PM, OQ
<overlordq(a)gmail.com> wrote:
So the testsuite only runs on merged code and not
pending-merge? That
sounds like a large oversight.
Picture in your mind every branch pending merge for every extension in
gerrit. Imagine how many of those branches are eventually abandoned,
imagine how many patch sets each receives, imagine how many times each gets
rebased.
And even if we had such tests, they would not have exposed today's issue.
We run UI-level regression tests against a model of the Wikipedia cluster
on beta labs running the master branch *exactly* so that we can expose
cross-repo problems, configuration problems, etc. before they go to
production.
Today's issue was hardly unique. Just one week ago our tests picked up an
entirely unrelated but similarly surprising issue that had the
MobileFrontend team scrambling on a Thursday morning:
https://bugzilla.wikimedia.org/show_bug.cgi?id=62004. We stop bugs *all
the time* this way.
This is hardly an "oversight". These tests and these test environments are
very carefully designed to expose exactly the kind of issues that they
expose. They have saved us an extraordinary amount of pain by preventing
bugs released to production.
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l