[QA] [WikimediaMobile] No one cares about browser tests [Re: MobileFrontend QA job]

[Antoine Musso]

On 03/04/15 11:30, Joaquin Oltra Hernandez wrote:
> Personally, I love them, but more often than not when I run them a few
> things happen that make me sad:
>
>   * Tests take too long to run
>       o Even on Gather, which doesn't have too many tests, it takes a
>         really long time, which discourages running all tests every time.
>   * Tests don't block jenkins merges
>       o This would be key for me. Do this. Block jenkins. Then we'll be
>         forced to make it faster, make better browser tests, and
>         everybody will have to care because they will run and block merges.
>
> So there's a few key items that I would love to see:
>
>   * Improve performance (needs magnitudes improvements, a lot of work on
>     different fronts)
>   * Fully support headless browsers like phantom (they randomly break
>     with timeout errors and other problems, but are the faster/painless
>     ways of running the tests)
>   * Run the browser tests (or a smoke subset) on jenkins patches as a
>     voting job. Crucial for making everybody care about the tests and to
>     stop regressions.

Hello,

There is definitely an issue with the running time. A month or so ago I 
have noticed that each scenario takes 30-40 seconds during 
initialization because of network round trip with the VM provider 
SauceLabs: https://phabricator.wikimedia.org/T92613

There are most probably more optimization to follow.

The scenario are run using cucumber and serially, by running scenarii in 
parallel consuming a pool of VM would dramatically speed up the run. 
Alas cucumber does not support that out of the box and nobody committed 
to figure out a way to run them in parallel.


To have browser tests blocking merges (via a job that runs on CR+2), we 
need to setup some MediaWiki stack.  We did some experiment early 2014 
but were not quite happy with the result and gave up.
I would like to eventually come up with a solution such as MediaWiki 
Vagrant which would let us boot up a MediaWiki env suitable for testing 
that would have the +2 ed patch in it.  But that require some extra work.
The idea is not abandoned, a preprequisite is to have one time use 
virtual machine booted by our CI infrastructure.  That is the project I 
am currently actively working on with ops.


Timo pointed out that phantomjs is using some old version of Webkit that 
is no more matching any browser currently in use.  We can though use 
XVFB (X Virtual Frame Buffer) which emulate X Window in memory and run 
Chromium or Firefox in it.  That is quite fast. The QUnit tests that 
uses Karma have been migrated to it afaik.

Smoke test / subset is great.  I had a discussion with Zeljkof and he 
mentioned there is no point for browser tests to test everything, just 
need a few end user scenario.  The rest should be tested via integration 
tests.


If we could get developers from over teams to help on speeding up 
cucumber that would be already a huge improvement!

cheers,

-- 
Antoine Musso