On Wed, Nov 26, 2014 at 8:01 PM, Joaquin Oltra Hernandez jhernandez@wikimedia.org wrote:

TLDR:

phantomjs cucumber tests ~20% faster that firefox (100% less annoying) phantomjs cucumber tests have more failing tests than firefox cucumber tests have tags.

we should tag important/fast tests, and run them more often in dev

I've been experimenting with the browser tests to see if it would make any difference running them headlessly or w/ phantomjs in speed of execution.

First, in OSX it is not supported running the cucumber tests in headless mode (env var HEADLESS=true with any browser), so I haven't been able to time that.

Try BROWSER=phantomjs I'm not sure if HEADLESS=true is needed any more..

When running the cucumber tests on my machine, my results have been 19m30s for Firefox, and 15m for phantomjs.

It is not a huge improvement, but still, around 23% speed up, so it is worth it. Also, the browser window is not stealing focus from you while you work every time a new test launches, so I would say that is a major win, at least for me.

I'll investigate a bit more since I get more failing tests in phantom js than with firefox (28 vs 13) and also why I'm getting all those failing tests even in firefox. If anybody has faced this issues, I'd love some help (https://gist.github.com/joakin/853176ebf309798085f6)

About executing an important subset, as we discussed, browser tests have tags both for features and scenarios, and you can execute only tests with a certain tag by doing bundle exec cucumber --tags @tagname for example. You can see an example of tagging feature and scenario in mainmenu.feature for example:

@chrome @en.m.wikipedia.beta.wmflabs.org @firefox @test2.m.wikipedia.org @vagrant Feature: Menus open correct page for anonymous users ... @extension-geodata Scenario: Nearby link in menu ...

I think it would benefit us a lot to tag both important and fast tests, so that we run them more often in development and catch more regressions earlier.

I agree. @fast and @important tags would be useful.

To run browser tests in mobilefrontend with phantomjs installed just do BROWSER=phantomjs make cucumber

More info:

https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_tes...

I've cc'ed the qa and mobile-l public mailing lists as I think this is a useful discussion and will gather more expertise :).