[QA] Findings about browser tests (cucumber - phantomjs)

[Jon Robson]

On Wed, Nov 26, 2014 at 8:01 PM, Joaquin Oltra Hernandez
<jhernandez at wikimedia.org> wrote:
> TLDR:
>
> phantomjs cucumber tests ~20% faster that firefox (100% less annoying)
> phantomjs cucumber tests have more failing tests than firefox
> cucumber tests have tags.
>
> we should tag important/fast tests, and run them more often in dev
>
> I've been experimenting with the browser tests to see if it would make any
> difference running them headlessly or w/ phantomjs in speed of execution.
>
> First, in OSX it is not supported running the cucumber tests in headless
> mode (env var HEADLESS=true with any browser), so I haven't been able to
> time that.

Try BROWSER=phantomjs
I'm not sure if HEADLESS=true is needed any more..

>
> When running the cucumber tests on my machine, my results have been 19m30s
> for Firefox, and 15m for phantomjs.
>
> It is not a huge improvement, but still, around 23% speed up, so it is worth
> it. Also, the browser window is not stealing focus from you while you work
> every time a new test launches, so I would say that is a major win, at least
> for me.
>
> I'll investigate a bit more since I get more failing tests in phantom js
> than with firefox (28 vs 13) and also why I'm getting all those failing
> tests even in firefox. If anybody has faced this issues, I'd love some help
> (https://gist.github.com/joakin/853176ebf309798085f6)
>
> About executing an important subset, as we discussed, browser tests have
> tags both for features and scenarios, and you can execute only tests with a
> certain tag by doing bundle exec cucumber --tags @tagname for example. You
> can see an example of tagging feature and scenario in mainmenu.feature for
> example:
>
> @chrome @en.m.wikipedia.beta.wmflabs.org @firefox @test2.m.wikipedia.org
> @vagrant
> Feature: Menus open correct page for anonymous users
> ...
>   @extension-geodata
>   Scenario: Nearby link in menu
>   ...
>
> I think it would benefit us a lot to tag both important and fast tests, so
> that we run them more often in development and catch more regressions
> earlier.

I agree. @fast and @important tags would be useful.

>
> To run browser tests in mobilefrontend with phantomjs installed just do
> BROWSER=phantomjs make cucumber
>
> More info:
>
> https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_tests#Running_the_tests

I've cc'ed the qa and mobile-l public mailing lists as I think this is
a useful discussion and will gather more expertise :).