[QA] [WikimediaMobile] Visual regression testing
Juliusz Gonera
jgonera at wikimedia.org
Thu May 22 18:36:12 UTC 2014
Sure, but that doesn't blend with current Selenium tests, does it?
Also, I'm not sure if tests should pass on significant difference in
screenshots. I agree that getting this right to avoid false alarms is
tricky, but it would be useful. Otherwise, I'm afraid nobody will pay
attention.
On 05/08/2014 11:57 AM, Krinkle wrote:
> Could use something like pdiff:
> https://bugzilla.wikimedia.org/show_bug.cgi?id=62633
>
> Would compare the screenshot during the 'test' pipeline, against the
> screenshot from the last post-merge for the same branch.
> Wouldn't and shouldn't give you test passing / test failing. It would
> instead report to Gerrit with how much (if) there is different
> (percentage/ratio or something), and you can then follow the link to
> look at it yourself. Especially if you didn't intend for visual changes.
>
> Been using it in some smaller projects and has been very helpful and
> time saving.
>
> — Krinkle
>
> On 6 May 2014, at 22:43, Juliusz Gonera <jgonera at wikimedia.org
> <mailto:jgonera at wikimedia.org>> wrote:
>
>> Agreed, but it would be good for this mechanism to be as decoupled
>> from the tests themselves as possible. This way it can be reused and
>> improved by other people (outside of WMF).
>>
>>
>> On 04/22/2014 08:58 AM, Chris McMahon wrote:
>>>
>>> Yes, but I think it might be possible to share the underlying
>>> mechanism of taking screen shots among all the projects that want to
>>> do that, for whatever purposes they have.
>>>
>>> -Chris
>>>
>>>
>>> On Tue, Apr 22, 2014 at 8:53 AM, Jon Robson <jrobson at wikimedia.org
>>> <mailto:jrobson at wikimedia.org>> wrote:
>>>
>>> This wouldn't just be when a test fails. It would be used to detect
>>> regressions in appearance at given steps....
>>> e.g. Given I go to the login page When I trigger an error Then I see
>>> an error message And the login page looks exactly like it was a week
>>> ago.
>>>
>>>
>>> On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon
>>> <cmcmahon at wikimedia.org <mailto:cmcmahon at wikimedia.org>> wrote:
>>> >
>>> >
>>> > Language people also working on screen shots:
>>> > https://gerrit.wikimedia.org/r/#/c/127899/
>>> >
>>> >
>>> > On Mon, Apr 21, 2014 at 2:25 PM, Juliusz Gonera
>>> <jgonera at wikimedia.org <mailto:jgonera at wikimedia.org>>
>>> > wrote:
>>> >>
>>> >> Nope, it's an early prototype. It basically only makes an
>>> updated Cucumber
>>> >> test fail if new screenshot differs from reference screenshot
>>> by more than
>>> >> X%. Adding visual comparison of two screenshots would be the
>>> next step and
>>> >> would not be too hard. I might work on it one evening this week.
>>> >>
>>> >>
>>> >>
>>> >> On 04/21/2014 11:42 AM, Tomasz Finc wrote:
>>> >>>
>>> >>> Great. Eager to see us test with this.
>>> >>>
>>> >>> Do we have any of its early screenshot comparisons up?
>>> >>>
>>> >>> --tomasz
>>> >>>
>>> >>> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera
>>> <jgonera at wikimedia.org <mailto:jgonera at wikimedia.org>>
>>> >>> wrote:
>>> >>>>
>>> >>>> I conducted a little bit of research on visual regression
>>> testing,
>>> >>>> mainly
>>> >>>> for the mobile team for now, but it could be easily reused
>>> for other
>>> >>>> teams.
>>> >>>> I had a look at three existing solutions that seem somewhat
>>> popular and
>>> >>>> are
>>> >>>> actively developed:
>>> >>>>
>>> >>>> * Wraith (https://github.com/BBC-News/wraith)
>>> >>>> * PhantomCSS (https://github.com/Huddle/PhantomCSS)
>>> >>>> * Huxley (https://github.com/facebook/huxley)
>>> >>>>
>>> >>>> They all have their own pros and cons, but in my opinion
>>> they share one
>>> >>>> important disadvantage: they can't be easily integrated
>>> with our current
>>> >>>> browser testing setup. For all the aforementioned tools we
>>> would have to
>>> >>>> create from scratch a completely separate set of tests just
>>> for visual
>>> >>>> regression testing instead of extending our existing
>>> browser tests.
>>> >>>>
>>> >>>> I spent a few hours in my spare time tinkering with an
>>> alternative idea
>>> >>>> which would enable us to add visual regression testing to our
>>> >>>> Cucumber/Watir
>>> >>>> tests. The result is a small prototype available at
>>> >>>> https://github.com/jgonera/photographer. There is no docs
>>> or anything
>>> >>>> yet,
>>> >>>> but I prepared a simple demo patch for MobileFrontend:
>>> >>>> https://gerrit.wikimedia.org/r/#/c/126878/.
>>> >>>>
>>> >>>> The idea is to add a new method for Cucumber steps (snap)
>>> that takes a
>>> >>>> screenshot of the current browser state and compares it with a
>>> >>>> screenshot
>>> >>>> taken in one of the previous test runs. To update
>>> screenshots that are
>>> >>>> used
>>> >>>> as a reference you run tests with env var
>>> PHOTOGRAPHER=update. If newly
>>> >>>> taken screenshot differs by too many pixels from an old
>>> one, the test
>>> >>>> will
>>> >>>> fail.
>>> >>>>
>>> >>>> It's still only an early prototype, but I'd appreciate any
>>> comments
>>> >>>> about
>>> >>>> this idea.
>>> >>>>
>>> >>>> --
>>> >>>> Juliusz
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Mobile-l mailing list
>>> >>>> Mobile-l at lists.wikimedia.org
>>> <mailto:Mobile-l at lists.wikimedia.org>
>>> >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>>> >>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> QA mailing list
>>> >> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>>> >> https://lists.wikimedia.org/mailman/listinfo/qa
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > QA mailing list
>>> > QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>>> > https://lists.wikimedia.org/mailman/listinfo/qa
>>> >
>>>
>>> _______________________________________________
>>> QA mailing list
>>> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>>> https://lists.wikimedia.org/mailman/listinfo/qa
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> QA mailing list
>>> QA at lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/qa
>>
>> _______________________________________________
>> QA mailing list
>> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>> https://lists.wikimedia.org/mailman/listinfo/qa
>
>
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140522/4fde1e6f/attachment.html>
More information about the QA
mailing list