[QA] [reading-wmf] Recently failing browser tests

[Dan Duvall]

On Thu, Jun 4, 2015 at 10:42 AM, Jon Robson <jrobson at wikimedia.org> wrote:

> Cc'ing QA so they can see my latest reply.
>
> On Thu, Jun 4, 2015 at 10:42 AM, Jon Robson <jrobson at wikimedia.org> wrote:
>
>> Gather tests take around 7 min 21s to run [1]
>> Smoke tests 4 mins [2]
>> These would be the initial target.
>>
>
That seems like a more reasonable target albeit with proportionally less
immediate payoff. Once we have the underlying CI isolation work done,[1] we
should definitely shoot for pre-merge job for Gather and other repos with
well-defined performant smoke tests.

[1] https://phabricator.wikimedia.org/T47499

Yesterday 3 patches got merged that had been sitting in Gerrit for approx 3
>> days that caused the test to break in both Gather and MobileFrontend.
>> Had we run the smoke tests on these tests at some point during those 3
>> days we would have been able to fix them pre-merging.
>>
>
For the proposed setup to help in such cases, the job will need to be
defined in JJB and scheduled to run via the gate-and-submit pipeline
(pre-merge). At this point the job will be fully integrated into CI and
will need to be quite fast and reliable lest it promote force merges.


Personally I am fine with the delay and running the test jobs one at a
>> time. I see this as a low risk useful grass routes experiment by a
>> smaller team that can be fed up to the rest of the Foundation and benefit
>> all teams.  I don't see this as duplicating efforts rather than
>> investigating something we know we all want but on a smaller scale.
>>
>
Yes, but this isn't a grassroots effort. It's a time commitment from staff
of WMF and we need to consider the cost (and technical debt) we're
incurring. (Unless it is being done on volunteer time and, in that case,
sorry for the misunderstanding.) I would rather see time commitments for
figuring out how to fill the void in process that it seems we're lacking
regarding test ownership and workflow around analyzing failures, filing
tasks, and fixing broken tests.

In other words, the relatively small amount of time you and I have spent
over the past couple days has been relatively painless and yielded tangible
payoff.[2] I'd like to see that kind of time investment be regular and
explicitly promoted by leadership.

[2]
https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/707/


-- 
Dan Duvall
Automation Engineer
Wikimedia Foundation <http://wikimediafoundation.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20150604/2265419a/attachment-0001.html>