[QA] [WikimediaMobile] Visual regression testing

[Juliusz Gonera]

Nope, it's an early prototype. It basically only makes an updated 
Cucumber test fail if new screenshot differs from reference screenshot 
by more than X%. Adding visual comparison of two screenshots would be 
the next step and would not be too hard. I might work on it one evening 
this week.


On 04/21/2014 11:42 AM, Tomasz Finc wrote:
> Great. Eager to see us test with this.
>
> Do we have any of its early screenshot comparisons up?
>
> --tomasz
>
> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera <jgonera at wikimedia.org> wrote:
>> I conducted a little bit of research on visual regression testing, mainly
>> for the mobile team for now, but it could be easily reused for other teams.
>> I had a look at three existing solutions that seem somewhat popular and are
>> actively developed:
>>
>> * Wraith (https://github.com/BBC-News/wraith)
>> * PhantomCSS (https://github.com/Huddle/PhantomCSS)
>> * Huxley (https://github.com/facebook/huxley)
>>
>> They all have their own pros and cons, but in my opinion they share one
>> important disadvantage: they can't be easily integrated with our current
>> browser testing setup. For all the aforementioned tools we would have to
>> create from scratch a completely separate set of tests just for visual
>> regression testing instead of extending our existing browser tests.
>>
>> I spent a few hours in my spare time tinkering with an alternative idea
>> which would enable us to add visual regression testing to our Cucumber/Watir
>> tests. The result is a small prototype available at
>> https://github.com/jgonera/photographer. There is no docs or anything yet,
>> but I prepared a simple demo patch for MobileFrontend:
>> https://gerrit.wikimedia.org/r/#/c/126878/.
>>
>> The idea is to add a new method for Cucumber steps (snap) that takes a
>> screenshot of the current browser state and compares it with a screenshot
>> taken in one of the previous test runs. To update screenshots that are used
>> as a reference you run tests with env var PHOTOGRAPHER=update. If newly
>> taken screenshot differs by too many pixels from an old one, the test will
>> fail.
>>
>> It's still only an early prototype, but I'd appreciate any comments about
>> this idea.
>>
>> --
>> Juliusz
>>
>> _______________________________________________
>> Mobile-l mailing list
>> Mobile-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/mobile-l