On Thu, Mar 6, 2014 at 6:07 PM, OQ overlordq@gmail.com wrote:
So the testsuite only runs on merged code and not pending-merge? That sounds like a large oversight.
Picture in your mind every branch pending merge for every extension in gerrit. Imagine how many of those branches are eventually abandoned, imagine how many patch sets each receives, imagine how many times each gets rebased.
And even if we had such tests, they would not have exposed today's issue.
We run UI-level regression tests against a model of the Wikipedia cluster on beta labs running the master branch *exactly* so that we can expose cross-repo problems, configuration problems, etc. before they go to production.
Today's issue was hardly unique. Just one week ago our tests picked up an entirely unrelated but similarly surprising issue that had the MobileFrontend team scrambling on a Thursday morning: https://bugzilla.wikimedia.org/show_bug.cgi?id=62004. We stop bugs *all the time* this way.
This is hardly an "oversight". These tests and these test environments are very carefully designed to expose exactly the kind of issues that they expose. They have saved us an extraordinary amount of pain by preventing bugs released to production.