Re: [WikimediaMobile] Findings about browser tests (cucumber - phantomjs)

TLDR: phantomjs cucumber tests ~20% faster that firefox (100% less annoying) phantomjs cucumber tests have more failing tests than firefox cucumber tests have tags. we should tag important/fast tests, and run them more often in dev I've been experimenting with the browser tests to see if it would make any difference running them headlessly or w/ phantomjs in speed of execution. First, in OSX it is not supported running the cucumber tests in headless mode (env var HEADLESS=true with any browser), so I haven't been able to time that.

When running the cucumber tests on my machine, my results have been 19m30s for Firefox, and 15m for phantomjs. It is not a huge improvement, but still, around 23% speed up, so it is worth it. Also, the browser window is not stealing focus from you while you work every time a new test launches, so I would say that is a major win, at least for me. I'll investigate a bit more since I get more failing tests in phantom js than with firefox (28 vs 13) and also why I'm getting all those failing tests even in firefox. If anybody has faced this issues, I'd love some help (https://gist.github.com/joakin/853176ebf309798085f6) About executing an important subset, as we discussed, browser tests have tags both for features and scenarios, and you can execute only tests with a certain tag by doing bundle exec cucumber --tags @tagname for example. You can see an example of tagging feature and scenario in mainmenu.feature for example: @chrome @en.m.wikipedia.beta.wmflabs.org @firefox @test2.m.wikipedia.org @vagrant Feature: Menus open correct page for anonymous users ... @extension-geodata Scenario: Nearby link in menu ... I think it would benefit us a lot to tag both important and fast tests, so that we run them more often in development and catch more regressions earlier.

To run browser tests in mobilefrontend with phantomjs installed just do BROWSER=phantomjs make cucumber More info: https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_te…