[QA] [WikimediaMobile] Visual regression testing
Juliusz Gonera
jgonera at wikimedia.org
Thu May 22 18:34:24 UTC 2014
Can you post a link to more info?
On 05/08/2014 10:36 PM, Vikas Yaligar wrote:
> Hello,
>
> I am Vikas, GSOC student working on Automatic cross-language
> screenshots for user documentation. My GSOC proposal was:
>
> 1. To tag the scenarios whose screenshots are required.
>
> 2. Have an after hook for that tag, which takes the screenshot.
>
> 3. Also in after hook I can perform any cropping.
>
> 4. The cropped images will be uploaded in wikimedia commons.
>
> The above steps are run as a new jenkins job for each language in
> order to get cross-language screenshot.
>
>
>
> On Thu, May 8, 2014 at 3:27 PM, Krinkle <krinklemail at gmail.com
> <mailto:krinklemail at gmail.com>> wrote:
>
> Could use something like pdiff:
> https://bugzilla.wikimedia.org/show_bug.cgi?id=62633
>
> Would compare the screenshot during the 'test' pipeline, against
> the screenshot from the last post-merge for the same branch.
> Wouldn't and shouldn't give you test passing / test failing. It
> would instead report to Gerrit with how much (if) there is
> different (percentage/ratio or something), and you can then follow
> the link to look at it yourself. Especially if you didn't intend
> for visual changes.
>
> Been using it in some smaller projects and has been very helpful
> and time saving.
>
> — Krinkle
>
> On 6 May 2014, at 22:43, Juliusz Gonera <jgonera at wikimedia.org
> <mailto:jgonera at wikimedia.org>> wrote:
>
>> Agreed, but it would be good for this mechanism to be as
>> decoupled from the tests themselves as possible. This way it can
>> be reused and improved by other people (outside of WMF).
>>
>>
>> On 04/22/2014 08:58 AM, Chris McMahon wrote:
>>>
>>> Yes, but I think it might be possible to share the underlying
>>> mechanism of taking screen shots among all the projects that
>>> want to do that, for whatever purposes they have.
>>>
>>> -Chris
>>>
>>>
>>> On Tue, Apr 22, 2014 at 8:53 AM, Jon Robson
>>> <jrobson at wikimedia.org <mailto:jrobson at wikimedia.org>> wrote:
>>>
>>> This wouldn't just be when a test fails. It would be used to
>>> detect
>>> regressions in appearance at given steps....
>>> e.g. Given I go to the login page When I trigger an error
>>> Then I see
>>> an error message And the login page looks exactly like it
>>> was a week
>>> ago.
>>>
>>>
>>> On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon
>>> <cmcmahon at wikimedia.org <mailto:cmcmahon at wikimedia.org>> wrote:
>>> >
>>> >
>>> > Language people also working on screen shots:
>>> > https://gerrit.wikimedia.org/r/#/c/127899/
>>> >
>>> >
>>> > On Mon, Apr 21, 2014 at 2:25 PM, Juliusz Gonera
>>> <jgonera at wikimedia.org <mailto:jgonera at wikimedia.org>>
>>> > wrote:
>>> >>
>>> >> Nope, it's an early prototype. It basically only makes an
>>> updated Cucumber
>>> >> test fail if new screenshot differs from reference
>>> screenshot by more than
>>> >> X%. Adding visual comparison of two screenshots would be
>>> the next step and
>>> >> would not be too hard. I might work on it one evening
>>> this week.
>>> >>
>>> >>
>>> >>
>>> >> On 04/21/2014 11:42 AM, Tomasz Finc wrote:
>>> >>>
>>> >>> Great. Eager to see us test with this.
>>> >>>
>>> >>> Do we have any of its early screenshot comparisons up?
>>> >>>
>>> >>> --tomasz
>>> >>>
>>> >>> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz Gonera
>>> <jgonera at wikimedia.org <mailto:jgonera at wikimedia.org>>
>>> >>> wrote:
>>> >>>>
>>> >>>> I conducted a little bit of research on visual
>>> regression testing,
>>> >>>> mainly
>>> >>>> for the mobile team for now, but it could be easily
>>> reused for other
>>> >>>> teams.
>>> >>>> I had a look at three existing solutions that seem
>>> somewhat popular and
>>> >>>> are
>>> >>>> actively developed:
>>> >>>>
>>> >>>> * Wraith (https://github.com/BBC-News/wraith)
>>> >>>> * PhantomCSS (https://github.com/Huddle/PhantomCSS)
>>> >>>> * Huxley (https://github.com/facebook/huxley)
>>> >>>>
>>> >>>> They all have their own pros and cons, but in my
>>> opinion they share one
>>> >>>> important disadvantage: they can't be easily integrated
>>> with our current
>>> >>>> browser testing setup. For all the aforementioned tools
>>> we would have to
>>> >>>> create from scratch a completely separate set of tests
>>> just for visual
>>> >>>> regression testing instead of extending our existing
>>> browser tests.
>>> >>>>
>>> >>>> I spent a few hours in my spare time tinkering with an
>>> alternative idea
>>> >>>> which would enable us to add visual regression testing
>>> to our
>>> >>>> Cucumber/Watir
>>> >>>> tests. The result is a small prototype available at
>>> >>>> https://github.com/jgonera/photographer. There is no
>>> docs or anything
>>> >>>> yet,
>>> >>>> but I prepared a simple demo patch for MobileFrontend:
>>> >>>> https://gerrit.wikimedia.org/r/#/c/126878/.
>>> >>>>
>>> >>>> The idea is to add a new method for Cucumber steps
>>> (snap) that takes a
>>> >>>> screenshot of the current browser state and compares it
>>> with a
>>> >>>> screenshot
>>> >>>> taken in one of the previous test runs. To update
>>> screenshots that are
>>> >>>> used
>>> >>>> as a reference you run tests with env var
>>> PHOTOGRAPHER=update. If newly
>>> >>>> taken screenshot differs by too many pixels from an old
>>> one, the test
>>> >>>> will
>>> >>>> fail.
>>> >>>>
>>> >>>> It's still only an early prototype, but I'd appreciate
>>> any comments
>>> >>>> about
>>> >>>> this idea.
>>> >>>>
>>> >>>> --
>>> >>>> Juliusz
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Mobile-l mailing list
>>> >>>> Mobile-l at lists.wikimedia.org
>>> <mailto:Mobile-l at lists.wikimedia.org>
>>> >>>> https://lists.wikimedia.org/mailman/listinfo/mobile-l
>>> >>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> QA mailing list
>>> >> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>>> >> https://lists.wikimedia.org/mailman/listinfo/qa
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > QA mailing list
>>> > QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>>> > https://lists.wikimedia.org/mailman/listinfo/qa
>>> >
>>>
>>> _______________________________________________
>>> QA mailing list
>>> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>>> https://lists.wikimedia.org/mailman/listinfo/qa
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> QA mailing list
>>> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>>> https://lists.wikimedia.org/mailman/listinfo/qa
>>
>> _______________________________________________
>> QA mailing list
>> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
>> https://lists.wikimedia.org/mailman/listinfo/qa
>
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org <mailto:QA at lists.wikimedia.org>
> https://lists.wikimedia.org/mailman/listinfo/qa
>
>
>
>
> --
> Vikas S Yaligar
> (9481543789)
> IT Department
> NITK Surathkal
>
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140522/4374ac38/attachment-0001.html>
More information about the QA
mailing list