I conducted a little bit of research on visual regression testing,
mainly for the mobile team for now, but it could be easily reused for
other teams. I had a look at three existing solutions that seem somewhat
popular and are actively developed:
* Wraith (
https://github.com/BBC-News/wraith)
* PhantomCSS (
https://github.com/Huddle/PhantomCSS)
* Huxley (
https://github.com/facebook/huxley)
They all have their own pros and cons, but in my opinion they share one
important disadvantage: they can't be easily integrated with our current
browser testing setup. For all the aforementioned tools we would have to
create from scratch a completely separate set of tests just for visual
regression testing instead of extending our existing browser tests.
I spent a few hours in my spare time tinkering with an alternative idea
which would enable us to add visual regression testing to our
Cucumber/Watir tests. The result is a small prototype available at
https://github.com/jgonera/photographer. There is no docs or anything
yet, but I prepared a simple demo patch for MobileFrontend:
https://gerrit.wikimedia.org/r/#/c/126878/.
The idea is to add a new method for Cucumber steps (snap) that takes a
screenshot of the current browser state and compares it with a
screenshot taken in one of the previous test runs. To update screenshots
that are used as a reference you run tests with env var
PHOTOGRAPHER=update. If newly taken screenshot differs by too many
pixels from an old one, the test will fail.
It's still only an early prototype, but I'd appreciate any comments
about this idea.
--
Juliusz