[QA] [reading-wmf] Recently failing browser tests

[Gergo Tisza]

On Thu, Jun 4, 2015 at 10:34 AM, Dan Duvall <dduvall at wikimedia.org> wrote:

> In other words, there were 12 non-merge commits made yesterday to
> MobileFrontend alone; that's around 9 hours of run time for the tests to
> complete which, on a single instance, will have to be run in serial to
> ensure the tests don't interfere with one another; that essentially
> nullifies the more expedient feedback that you're hoping to gain.
>
> You could scale up the number of instances in the pool, but at that point
> the ad hoc setup will be duplicating a large portion of our shared CI
> infrastructure, which brings me to the next unanswered question: Who is
> going to maintain this setup? I don't mean to be a naysayer here, but I
> worry about the effects of implementing such a complex—and seemingly
> volatile setup—without a clear understanding within Infrastructure or
> Reading about its real value or long-term maintenance burden. Perhaps this
> is misplaced anxiety, but this screams tech debt incarnate.
>

A separate instance (or full serialization) for each test seems like an
enormous waste of resources. At worst, each test needs a separate database
and can share the VM and run fully parallel; setting up a new database from
a dump of a known good state should be in the seconds range. Ideally, most
tests would be read-only and marked as such, and those could run on the
same DB.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20150604/510d0a97/attachment.html>