Re: [WikimediaMobile] Visual regression testing

I conducted a little bit of research on visual regression testing, mainly for the mobile team for now, but it could be easily reused for other teams. I had a look at three existing solutions that seem somewhat popular and are actively developed: * Wraith (https://github.com/BBC-News/wraith) * PhantomCSS (https://github.com/Huddle/PhantomCSS) * Huxley (https://github.com/facebook/huxley) They all have their own pros and cons, but in my opinion they share one important disadvantage: they can't be easily integrated with our current browser testing setup. For all the aforementioned tools we would have to create from scratch a completely separate set of tests just for visual regression testing instead of extending our existing browser tests. I spent a few hours in my spare time tinkering with an alternative idea which would enable us to add visual regression testing to our Cucumber/Watir tests. The result is a small prototype available at https://github.com/jgonera/photographer. There is no docs or anything yet, but I prepared a simple demo patch for MobileFrontend: https://gerrit.wikimedia.org/r/#/c/126878/. The idea is to add a new method for Cucumber steps (snap) that takes a screenshot of the current browser state and compares it with a screenshot taken in one of the previous test runs. To update screenshots that are used as a reference you run tests with env var PHOTOGRAPHER=update. If newly taken screenshot differs by too many pixels from an old one, the test will fail. It's still only an early prototype, but I'd appreciate any comments about this idea. -- Juliusz _______________________________________________ Mobile-l mailing list Mobile-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l