[QA] Cut testing time for Cirrus drastically

Nikolas Everett neverett at wikimedia.org
Thu Jan 30 02:57:04 UTC 2014


Yesterday I spent the time working on CirrusSearch's browser tests to
get them working well with parallel cucumber and phantomjs. That cut
the time to run all the tests from 24 minutes to 6.  That is super
sweet because Cirrus gets the vast majority of its test coverage from
these tests.

So here are the steps I took to get there.  You can probably do them
in any order if you want to follow my footsteps.  Anyway:
1.  Make sure all tests pass in PhantomJS.  Mostly, this shouldn't
require any work beyond installing PhantomJS, setting BROWSER_LABEL to
phantomjs, and running tests.
2.  Make sure your tests are in many small features rather than a few
huge ones.  This took me an hour or so because Cirrus had two huge
features that I broke into 18 or so small ones.
3.  Add <gem "parallel_tests"> to your gemfile and run bundler install.
4.  You are ready!  Yay.

Running tests is now a two step thing.  First run tests in parallel.
  bundle exec parallel_cucumber --nice -n 5 features/
Some of them will fail because, well, parallel_cucumber seems a bit
flaky.  Oh well.  You can rerun them like this:
  cat cucumber_failures.log | xargs bundle exec cucumber

I'm not sure how you'd do that in Windows but the above should work in
Linux and OSX.


Nik



More information about the QA mailing list