[QA] [WikimediaMobile] Visual regression testing

[Krinkle]

Could use something like pdiff:
	https://bugzilla.wikimedia.org/show_bug.cgi?id=62633

Would compare the screenshot during the 'test' pipeline, against the screenshot from the last post-merge for the same branch.
Wouldn't and shouldn't give you test passing / test failing. It would instead report to Gerrit with how much (if) there is different (percentage/ratio or something), and you can then follow the link to look at it yourself. Especially if you didn't intend for visual changes.

Been using it in some smaller projects and has been very helpful and time saving.

— Krinkle

On 6 May 2014, at 22:43, Juliusz Gonera <jgonera at wikimedia.org> wrote:

> Agreed, but it would be good for this mechanism to be as decoupled from the tests themselves as possible. This way it can be reused and improved by other people (outside of WMF).
> 
> 
> On 04/22/2014 08:58 AM, Chris McMahon wrote:
>> 
>> Yes, but I think it might be possible to share the underlying mechanism of taking screen shots among all the projects that want to do that, for whatever purposes they have.  
>> 
>> -Chris 
>> 
>> 
>> On Tue, Apr 22, 2014 at 8:53 AM, Jon Robson <jrobson at wikimedia.org> wrote:
>> This wouldn't just be when a test fails. It would be used to detect
>> regressions in appearance at given steps....
>> e.g. Given I go to the login page When I trigger an error Then I see
>> an error message And the login page looks exactly like it was a week
>> ago.
>> 
>> 
>> On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon <cmcmahon at wikimedia.org> wrote:
>> >
>> >
>> > Language people also working on screen shots:
>> > https://gerrit.wikimedia.org/r/#/c/127899/
>> >
>> >
>> > On Mon, Apr 21, 2014 at 2:25 PM, Juliusz Gonera <jgonera at wikimedia.org>
>> > wrote:
>> >>
>> >> Nope, it's an early prototype. It basically only makes an updated Cucumber
>> >> test fail if new screenshot differs from reference screenshot by more than
>> >> X%. Adding visual comparison of two screenshots would be the next step and
>> >> would not be too hard. I might work on it one evening this week.
>> >>
>> >>
>> >>
>> >> On 04/21/2014 11:42 AM, Tomasz Finc wrote:
>> >>>
>> >>> Great. Eager to see us test with this.
>> >>>
>> >>> Do we have any of its early screenshot comparisons up?
>> >>>
>> >>> --tomasz
>> >>>
>> >>> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera <jgonera at wikimedia.org>
>> >>> wrote:
>> >>>>
>> >>>> I conducted a little bit of research on visual regression testing,
>> >>>> mainly
>> >>>> for the mobile team for now, but it could be easily reused for other
>> >>>> teams.
>> >>>> I had a look at three existing solutions that seem somewhat popular and
>> >>>> are
>> >>>> actively developed:
>> >>>>
>> >>>> * Wraith (https://github.com/BBC-News/wraith)
>> >>>> * PhantomCSS (https://github.com/Huddle/PhantomCSS)
>> >>>> * Huxley (https://github.com/facebook/huxley)
>> >>>>
>> >>>> They all have their own pros and cons, but in my opinion they share one
>> >>>> important disadvantage: they can't be easily integrated with our current
>> >>>> browser testing setup. For all the aforementioned tools we would have to
>> >>>> create from scratch a completely separate set of tests just for visual
>> >>>> regression testing instead of extending our existing browser tests.
>> >>>>
>> >>>> I spent a few hours in my spare time tinkering with an alternative idea
>> >>>> which would enable us to add visual regression testing to our
>> >>>> Cucumber/Watir
>> >>>> tests. The result is a small prototype available at
>> >>>> https://github.com/jgonera/photographer. There is no docs or anything
>> >>>> yet,
>> >>>> but I prepared a simple demo patch for MobileFrontend:
>> >>>> https://gerrit.wikimedia.org/r/#/c/126878/.
>> >>>>
>> >>>> The idea is to add a new method for Cucumber steps (snap) that takes a
>> >>>> screenshot of the current browser state and compares it with a
>> >>>> screenshot
>> >>>> taken in one of the previous test runs. To update screenshots that are
>> >>>> used
>> >>>> as a reference you run tests with env var PHOTOGRAPHER=update. If newly
>> >>>> taken screenshot differs by too many pixels from an old one, the test
>> >>>> will
>> >>>> fail.
>> >>>>
>> >>>> It's still only an early prototype, but I'd appreciate any comments
>> >>>> about
>> >>>> this idea.
>> >>>>
>> >>>> --
>> >>>> Juliusz
>> >>>>
>> >>>> _______________________________________________
>> >>>> Mobile-l mailing list
>> >>>> Mobile-l at lists.wikimedia.org
>> >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> QA mailing list
>> >> QA at lists.wikimedia.org
>> >> https://lists.wikimedia.org/mailman/listinfo/qa
>> >
>> >
>> >
>> > _______________________________________________
>> > QA mailing list
>> > QA at lists.wikimedia.org
>> > https://lists.wikimedia.org/mailman/listinfo/qa
>> >
>> 
>> _______________________________________________
>> QA mailing list
>> QA at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/qa
>> 
>> 
>> 
>> _______________________________________________
>> QA mailing list
>> QA at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/qa
> 
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140508/b234338f/attachment.html>