[QA] [WikimediaMobile] Visual regression testing

Tue May 6 20:43:04 UTC 2014

Agreed, but it would be good for this mechanism to be as decoupled from 
the tests themselves as possible. This way it can be reused and improved 
by other people (outside of WMF).

On 04/22/2014 08:58 AM, Chris McMahon wrote:
>
> Yes, but I think it might be possible to share the underlying 
> mechanism of taking screen shots among all the projects that want to 
> do that, for whatever purposes they have.
>
> -Chris
>
>
> On Tue, Apr 22, 2014 at 8:53 AM, Jon Robson <jrobson at wikimedia.org 
> <mailto:jrobson at wikimedia.org>> wrote:
>
>     This wouldn't just be when a test fails. It would be used to detect
>     regressions in appearance at given steps....
>     e.g. Given I go to the login page When I trigger an error Then I see
>     an error message And the login page looks exactly like it was a week
>     ago.
>
>
>     On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon
>     <cmcmahon at wikimedia.org <mailto:cmcmahon at wikimedia.org>> wrote:
>     >
>     >
>     > Language people also working on screen shots:
>     > https://gerrit.wikimedia.org/r/#/c/127899/
>     >
>     >
>     > On Mon, Apr 21, 2014 at 2:25 PM, Juliusz Gonera
>     <jgonera at wikimedia.org <mailto:jgonera at wikimedia.org>>
>     > wrote:
>     >>
>     >> Nope, it's an early prototype. It basically only makes an
>     updated Cucumber
>     >> test fail if new screenshot differs from reference screenshot
>     by more than
>     >> X%. Adding visual comparison of two screenshots would be the
>     next step and
>     >> would not be too hard. I might work on it one evening this week.
>     >>
>     >>
>     >>
>     >> On 04/21/2014 11:42 AM, Tomasz Finc wrote:
>     >>>
>     >>> Great. Eager to see us test with this.
>     >>>
>     >>> Do we have any of its early screenshot comparisons up?
>     >>>
>     >>> --tomasz
>     >>>
>     >>> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera
>     <jgonera at wikimedia.org <mailto:jgonera at wikimedia.org>>
>     >>> wrote:
>     >>>>
>     >>>> I conducted a little bit of research on visual regression
>     testing,
>     >>>> mainly
>     >>>> for the mobile team for now, but it could be easily reused
>     for other
>     >>>> teams.
>     >>>> I had a look at three existing solutions that seem somewhat
>     popular and
>     >>>> are
>     >>>> actively developed:
>     >>>>
>     >>>> * Wraith (https://github.com/BBC-News/wraith)
>     >>>> * PhantomCSS (https://github.com/Huddle/PhantomCSS)
>     >>>> * Huxley (https://github.com/facebook/huxley)
>     >>>>
>     >>>> They all have their own pros and cons, but in my opinion they
>     share one
>     >>>> important disadvantage: they can't be easily integrated with
>     our current
>     >>>> browser testing setup. For all the aforementioned tools we
>     would have to
>     >>>> create from scratch a completely separate set of tests just
>     for visual
>     >>>> regression testing instead of extending our existing browser
>     tests.
>     >>>>
>     >>>> I spent a few hours in my spare time tinkering with an
>     alternative idea
>     >>>> which would enable us to add visual regression testing to our
>     >>>> Cucumber/Watir
>     >>>> tests. The result is a small prototype available at
>     >>>> https://github.com/jgonera/photographer. There is no docs or
>     anything
>     >>>> yet,
>     >>>> but I prepared a simple demo patch for MobileFrontend:
>     >>>> https://gerrit.wikimedia.org/r/#/c/126878/.
>     >>>>
>     >>>> The idea is to add a new method for Cucumber steps (snap)
>     that takes a
>     >>>> screenshot of the current browser state and compares it with a
>     >>>> screenshot
>     >>>> taken in one of the previous test runs. To update screenshots
>     that are
>     >>>> used
>     >>>> as a reference you run tests with env var
>     PHOTOGRAPHER=update. If newly
>     >>>> taken screenshot differs by too many pixels from an old one,
>     the test
>     >>>> will
>     >>>> fail.
>     >>>>
>     >>>> It's still only an early prototype, but I'd appreciate any
>     comments
>     >>>> about
>     >>>> this idea.
>     >>>>
>     >>>> --
>     >>>> Juliusz
>     >>>>
>     >>>> _______________________________________________
>     >>>> Mobile-l mailing list
>     >>>> Mobile-l at lists.wikimedia.org
>     <mailto:Mobile-l at lists.wikimedia.org>
>     >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>     >>
>     >>
>     >>
>     >> _______________________________________________
>     >> QA mailing list
>     >> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>     >> https://lists.wikimedia.org/mailman/listinfo/qa
>     >
>     >
>     >
>     > _______________________________________________
>     > QA mailing list
>     > QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>     > https://lists.wikimedia.org/mailman/listinfo/qa
>     >
>
>     _______________________________________________
>     QA mailing list
>     QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>     https://lists.wikimedia.org/mailman/listinfo/qa
>
>
>
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140506/869c874f/attachment-0001.html>