[QA] [reading-wmf] Recently failing browser tests

[Dan Duvall]

On Thu, Jun 4, 2015 at 11:22 AM, Gergo Tisza <gtisza at wikimedia.org> wrote:

> On Thu, Jun 4, 2015 at 10:34 AM, Dan Duvall <dduvall at wikimedia.org> wrote:
>
>> In other words, there were 12 non-merge commits made yesterday to
>> MobileFrontend alone; that's around 9 hours of run time for the tests to
>> complete which, on a single instance, will have to be run in serial to
>> ensure the tests don't interfere with one another; that essentially
>> nullifies the more expedient feedback that you're hoping to gain.
>>
>> You could scale up the number of instances in the pool, but at that point
>> the ad hoc setup will be duplicating a large portion of our shared CI
>> infrastructure, which brings me to the next unanswered question: Who is
>> going to maintain this setup? I don't mean to be a naysayer here, but I
>> worry about the effects of implementing such a complex—and seemingly
>> volatile setup—without a clear understanding within Infrastructure or
>> Reading about its real value or long-term maintenance burden. Perhaps this
>> is misplaced anxiety, but this screams tech debt incarnate.
>>
>
> A separate instance (or full serialization) for each test seems like an
> enormous waste of resources.
>

Mostly definitely. We've talked about using LXC (or perhaps Docker) to
achieve the isolation without having to spin up a new instance for each
build. The trick here is to achieve a setup where we can quickly and
cleanly initialize the workspace while still facilitating some level of
caching for expensive dependencies (composer, npm, bundler) outside of its
environment—containers seem like the right option for this.


> At worst, each test needs a separate database and can share the VM and run
> fully parallel; setting up a new database from a dump of a known good state
> should be in the seconds range. Ideally, most tests would be read-only and
> marked as such, and those could run on the same DB.
>

I'm not sure we'll ever be able to assume atomic read-only tests, maybe for
integration tests but not end-to-end tests—another reason why I think it's
important to differentiate our tests—which is why we need the isolation.


>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>
>


-- 
Dan Duvall
Automation Engineer
Wikimedia Foundation <http://wikimediafoundation.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20150604/9a3e8e26/attachment.html>