[QA] [WikimediaMobile] Visual regression testing

Thu May 8 20:36:44 UTC 2014

Hello,

I am Vikas, GSOC student working on Automatic cross-language screenshots
for user documentation. My GSOC proposal was:

1. To tag the scenarios whose screenshots are required.

2. Have an after hook for that tag, which takes the screenshot.

3. Also in after hook I can perform any cropping.

4. The cropped images will be uploaded in wikimedia commons.

The above steps are run as a new jenkins job for each language in order to
get cross-language screenshot.

On Thu, May 8, 2014 at 3:27 PM, Krinkle <krinklemail at gmail.com> wrote:

> Could use something like pdiff:
> https://bugzilla.wikimedia.org/show_bug.cgi?id=62633
>
> Would compare the screenshot during the 'test' pipeline, against the
> screenshot from the last post-merge for the same branch.
> Wouldn't and shouldn't give you test passing / test failing. It would
> instead report to Gerrit with how much (if) there is different
> (percentage/ratio or something), and you can then follow the link to look
> at it yourself. Especially if you didn't intend for visual changes.
>
> Been using it in some smaller projects and has been very helpful and time
> saving.
>
> -- Krinkle
>
> On 6 May 2014, at 22:43, Juliusz Gonera <jgonera at wikimedia.org> wrote:
>
>  Agreed, but it would be good for this mechanism to be as decoupled from
> the tests themselves as possible. This way it can be reused and improved by
> other people (outside of WMF).
>
>
> On 04/22/2014 08:58 AM, Chris McMahon wrote:
>
>
> Yes, but I think it might be possible to share the underlying mechanism of
> taking screen shots among all the projects that want to do that, for
> whatever purposes they have.
>
>  -Chris
>
>
> On Tue, Apr 22, 2014 at 8:53 AM, Jon Robson <jrobson at wikimedia.org> wrote:
>
>> This wouldn't just be when a test fails. It would be used to detect
>> regressions in appearance at given steps....
>> e.g. Given I go to the login page When I trigger an error Then I see
>> an error message And the login page looks exactly like it was a week
>> ago.
>>
>>
>> On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon <cmcmahon at wikimedia.org>
>> wrote:
>> >
>> >
>> > Language people also working on screen shots:
>> > https://gerrit.wikimedia.org/r/#/c/127899/
>> >
>> >
>> > On Mon, Apr 21, 2014 at 2:25 PM, Juliusz Gonera <jgonera at wikimedia.org>
>> > wrote:
>> >>
>> >> Nope, it's an early prototype. It basically only makes an updated
>> Cucumber
>> >> test fail if new screenshot differs from reference screenshot by more
>> than
>> >> X%. Adding visual comparison of two screenshots would be the next step
>> and
>> >> would not be too hard. I might work on it one evening this week.
>> >>
>> >>
>> >>
>> >> On 04/21/2014 11:42 AM, Tomasz Finc wrote:
>> >>>
>> >>> Great. Eager to see us test with this.
>> >>>
>> >>> Do we have any of its early screenshot comparisons up?
>> >>>
>> >>> --tomasz
>> >>>
>> >>> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera <
>> jgonera at wikimedia.org>
>> >>> wrote:
>> >>>>
>> >>>> I conducted a little bit of research on visual regression testing,
>> >>>> mainly
>> >>>> for the mobile team for now, but it could be easily reused for other
>> >>>> teams.
>> >>>> I had a look at three existing solutions that seem somewhat popular
>> and
>> >>>> are
>> >>>> actively developed:
>> >>>>
>> >>>> * Wraith (https://github.com/BBC-News/wraith)
>> >>>> * PhantomCSS (https://github.com/Huddle/PhantomCSS)
>> >>>> * Huxley (https://github.com/facebook/huxley)
>> >>>>
>> >>>> They all have their own pros and cons, but in my opinion they share
>> one
>> >>>> important disadvantage: they can't be easily integrated with our
>> current
>> >>>> browser testing setup. For all the aforementioned tools we would
>> have to
>> >>>> create from scratch a completely separate set of tests just for
>> visual
>> >>>> regression testing instead of extending our existing browser tests.
>> >>>>
>> >>>> I spent a few hours in my spare time tinkering with an alternative
>> idea
>> >>>> which would enable us to add visual regression testing to our
>> >>>> Cucumber/Watir
>> >>>> tests. The result is a small prototype available at
>> >>>> https://github.com/jgonera/photographer. There is no docs or
>> anything
>> >>>> yet,
>> >>>> but I prepared a simple demo patch for MobileFrontend:
>> >>>> https://gerrit.wikimedia.org/r/#/c/126878/.
>> >>>>
>> >>>> The idea is to add a new method for Cucumber steps (snap) that takes
>> a
>> >>>> screenshot of the current browser state and compares it with a
>> >>>> screenshot
>> >>>> taken in one of the previous test runs. To update screenshots that
>> are
>> >>>> used
>> >>>> as a reference you run tests with env var PHOTOGRAPHER=update. If
>> newly
>> >>>> taken screenshot differs by too many pixels from an old one, the test
>> >>>> will
>> >>>> fail.
>> >>>>
>> >>>> It's still only an early prototype, but I'd appreciate any comments
>> >>>> about
>> >>>> this idea.
>> >>>>
>> >>>> --
>> >>>> Juliusz
>> >>>>
>> >>>> _______________________________________________
>> >>>> Mobile-l mailing list
>> >>>> Mobile-l at lists.wikimedia.org
>> >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> QA mailing list
>> >> QA at lists.wikimedia.org
>> >> https://lists.wikimedia.org/mailman/listinfo/qa
>> >
>> >
>> >
>> > _______________________________________________
>> > QA mailing list
>> > QA at lists.wikimedia.org
>> > https://lists.wikimedia.org/mailman/listinfo/qa
>> >
>>
>> _______________________________________________
>> QA mailing list
>> QA at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/qa
>>
>
>
>
> _______________________________________________
> QA mailing listQA at lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/qa
>
>
>  _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>
>
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>
>

-- 
Vikas S Yaligar
(9481543789)
IT Department
NITK Surathkal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140509/5fa22e9a/attachment-0001.html>