On Wed, Nov 26, 2014 at 8:01 PM, Joaquin Oltra Hernandez
<jhernandez(a)wikimedia.org> wrote:
TLDR:
phantomjs cucumber tests ~20% faster that firefox (100% less annoying)
phantomjs cucumber tests have more failing tests than firefox
cucumber tests have tags.
we should tag important/fast tests, and run them more often in dev
I've been experimenting with the browser tests to see if it would make any
difference running them headlessly or w/ phantomjs in speed of execution.
First, in OSX it is not supported running the cucumber tests in headless
mode (env var HEADLESS=true with any browser), so I haven't been able to
time that.
Try BROWSER=phantomjs
I'm not sure if HEADLESS=true is needed any more..
When running the cucumber tests on my machine, my results have been 19m30s
for Firefox, and 15m for phantomjs.
It is not a huge improvement, but still, around 23% speed up, so it is worth
it. Also, the browser window is not stealing focus from you while you work
every time a new test launches, so I would say that is a major win, at least
for me.
I'll investigate a bit more since I get more failing tests in phantom js
than with firefox (28 vs 13) and also why I'm getting all those failing
tests even in firefox. If anybody has faced this issues, I'd love some help
(
https://gist.github.com/joakin/853176ebf309798085f6)
About executing an important subset, as we discussed, browser tests have
tags both for features and scenarios, and you can execute only tests with a
certain tag by doing bundle exec cucumber --tags @tagname for example. You
can see an example of tagging feature and scenario in mainmenu.feature for
example:
@chrome @en.m.wikipedia.beta.wmflabs.org @firefox @test2.m.wikipedia.org
@vagrant
Feature: Menus open correct page for anonymous users
...
@extension-geodata
Scenario: Nearby link in menu
...
I think it would benefit us a lot to tag both important and fast tests, so
that we run them more often in development and catch more regressions
earlier.
I agree. @fast and @important tags would be useful.
To run browser tests in mobilefrontend with phantomjs installed just do
BROWSER=phantomjs make cucumber
More info:
https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_te…
I've cc'ed the qa and mobile-l public mailing lists as I think this is
a useful discussion and will gather more expertise :).