[QA] [WikimediaMobile] Visual regression testing

[Jon Robson]

This wouldn't just be when a test fails. It would be used to detect
regressions in appearance at given steps....
e.g. Given I go to the login page When I trigger an error Then I see
an error message And the login page looks exactly like it was a week
ago.


On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon <cmcmahon at wikimedia.org> wrote:
>
>
> Language people also working on screen shots:
> https://gerrit.wikimedia.org/r/#/c/127899/
>
>
> On Mon, Apr 21, 2014 at 2:25 PM, Juliusz Gonera <jgonera at wikimedia.org>
> wrote:
>>
>> Nope, it's an early prototype. It basically only makes an updated Cucumber
>> test fail if new screenshot differs from reference screenshot by more than
>> X%. Adding visual comparison of two screenshots would be the next step and
>> would not be too hard. I might work on it one evening this week.
>>
>>
>>
>> On 04/21/2014 11:42 AM, Tomasz Finc wrote:
>>>
>>> Great. Eager to see us test with this.
>>>
>>> Do we have any of its early screenshot comparisons up?
>>>
>>> --tomasz
>>>
>>> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera <jgonera at wikimedia.org>
>>> wrote:
>>>>
>>>> I conducted a little bit of research on visual regression testing,
>>>> mainly
>>>> for the mobile team for now, but it could be easily reused for other
>>>> teams.
>>>> I had a look at three existing solutions that seem somewhat popular and
>>>> are
>>>> actively developed:
>>>>
>>>> * Wraith (https://github.com/BBC-News/wraith)
>>>> * PhantomCSS (https://github.com/Huddle/PhantomCSS)
>>>> * Huxley (https://github.com/facebook/huxley)
>>>>
>>>> They all have their own pros and cons, but in my opinion they share one
>>>> important disadvantage: they can't be easily integrated with our current
>>>> browser testing setup. For all the aforementioned tools we would have to
>>>> create from scratch a completely separate set of tests just for visual
>>>> regression testing instead of extending our existing browser tests.
>>>>
>>>> I spent a few hours in my spare time tinkering with an alternative idea
>>>> which would enable us to add visual regression testing to our
>>>> Cucumber/Watir
>>>> tests. The result is a small prototype available at
>>>> https://github.com/jgonera/photographer. There is no docs or anything
>>>> yet,
>>>> but I prepared a simple demo patch for MobileFrontend:
>>>> https://gerrit.wikimedia.org/r/#/c/126878/.
>>>>
>>>> The idea is to add a new method for Cucumber steps (snap) that takes a
>>>> screenshot of the current browser state and compares it with a
>>>> screenshot
>>>> taken in one of the previous test runs. To update screenshots that are
>>>> used
>>>> as a reference you run tests with env var PHOTOGRAPHER=update. If newly
>>>> taken screenshot differs by too many pixels from an old one, the test
>>>> will
>>>> fail.
>>>>
>>>> It's still only an early prototype, but I'd appreciate any comments
>>>> about
>>>> this idea.
>>>>
>>>> --
>>>> Juliusz
>>>>
>>>> _______________________________________________
>>>> Mobile-l mailing list
>>>> Mobile-l at lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>>
>>
>>
>> _______________________________________________
>> QA mailing list
>> QA at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/qa
>
>
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>