<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Could use something like pdiff:<div><span class="Apple-tab-span" style="white-space:pre"> </span><a href="https://bugzilla.wikimedia.org/show_bug.cgi?id=62633">https://bugzilla.wikimedia.org/show_bug.cgi?id=62633</a></div><div><br></div><div>Would compare the screenshot during the 'test' pipeline, against the screenshot from the last post-merge for the same branch.</div><div>Wouldn't and shouldn't give you test passing / test failing. It would instead report to Gerrit with how much (if) there is different (percentage/ratio or something), and you can then follow the link to look at it yourself. Especially if you didn't intend for visual changes.</div><div><br></div><div>Been using it in some smaller projects and has been very helpful and time saving.</div><div><br><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">— Krinkle</div>
</div>
<br><div><div>On 6 May 2014, at 22:43, Juliusz Gonera <<a href="mailto:jgonera@wikimedia.org">jgonera@wikimedia.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<div text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Agreed, but it would be good for this
mechanism to be as decoupled from the tests themselves as
possible. This way it can be reused and improved by other people
(outside of WMF).<br>
<br>
<br>
On 04/22/2014 08:58 AM, Chris McMahon wrote:<br>
</div>
<blockquote cite="mid:CAJohBHSt_9Rc9PwdY3wP+mGNLY1w62sqEoHa-dgmPOXT7_YhGA@mail.gmail.com" type="cite">
<div dir="ltr"><br>
<div>Yes, but I think it might be possible to share the
underlying mechanism of taking screen shots among all the
projects that want to do that, for whatever purposes they
have. </div>
<div><br>
</div>
<div>
-Chris </div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Apr 22, 2014 at 8:53 AM, Jon
Robson <span dir="ltr"><<a moz-do-not-send="true" href="mailto:jrobson@wikimedia.org" target="_blank">jrobson@wikimedia.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">This
wouldn't just be when a test fails. It would be used to
detect<br>
regressions in appearance at given steps....<br>
e.g. Given I go to the login page When I trigger an error
Then I see<br>
an error message And the login page looks exactly like it
was a week<br>
ago.<br>
<div class="HOEnZb">
<div class="h5"><br>
<br>
On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon <<a moz-do-not-send="true" href="mailto:cmcmahon@wikimedia.org">cmcmahon@wikimedia.org</a>>
wrote:<br>
><br>
><br>
> Language people also working on screen shots:<br>
> <a moz-do-not-send="true" href="https://gerrit.wikimedia.org/r/#/c/127899/" target="_blank">https://gerrit.wikimedia.org/r/#/c/127899/</a><br>
><br>
><br>
> On Mon, Apr 21, 2014 at 2:25 PM, Juliusz Gonera
<<a moz-do-not-send="true" href="mailto:jgonera@wikimedia.org">jgonera@wikimedia.org</a>><br>
> wrote:<br>
>><br>
>> Nope, it's an early prototype. It basically
only makes an updated Cucumber<br>
>> test fail if new screenshot differs from
reference screenshot by more than<br>
>> X%. Adding visual comparison of two screenshots
would be the next step and<br>
>> would not be too hard. I might work on it one
evening this week.<br>
>><br>
>><br>
>><br>
>> On 04/21/2014 11:42 AM, Tomasz Finc wrote:<br>
>>><br>
>>> Great. Eager to see us test with this.<br>
>>><br>
>>> Do we have any of its early screenshot
comparisons up?<br>
>>><br>
>>> --tomasz<br>
>>><br>
>>> On Fri, Apr 18, 2014 at 4:33 PM, Juliusz
Gonera <<a moz-do-not-send="true" href="mailto:jgonera@wikimedia.org">jgonera@wikimedia.org</a>><br>
>>> wrote:<br>
>>>><br>
>>>> I conducted a little bit of research on
visual regression testing,<br>
>>>> mainly<br>
>>>> for the mobile team for now, but it
could be easily reused for other<br>
>>>> teams.<br>
>>>> I had a look at three existing
solutions that seem somewhat popular and<br>
>>>> are<br>
>>>> actively developed:<br>
>>>><br>
>>>> * Wraith (<a moz-do-not-send="true" href="https://github.com/BBC-News/wraith" target="_blank">https://github.com/BBC-News/wraith</a>)<br>
>>>> * PhantomCSS (<a moz-do-not-send="true" href="https://github.com/Huddle/PhantomCSS" target="_blank">https://github.com/Huddle/PhantomCSS</a>)<br>
>>>> * Huxley (<a moz-do-not-send="true" href="https://github.com/facebook/huxley" target="_blank">https://github.com/facebook/huxley</a>)<br>
>>>><br>
>>>> They all have their own pros and cons,
but in my opinion they share one<br>
>>>> important disadvantage: they can't be
easily integrated with our current<br>
>>>> browser testing setup. For all the
aforementioned tools we would have to<br>
>>>> create from scratch a completely
separate set of tests just for visual<br>
>>>> regression testing instead of extending
our existing browser tests.<br>
>>>><br>
>>>> I spent a few hours in my spare time
tinkering with an alternative idea<br>
>>>> which would enable us to add visual
regression testing to our<br>
>>>> Cucumber/Watir<br>
>>>> tests. The result is a small prototype
available at<br>
>>>> <a moz-do-not-send="true" href="https://github.com/jgonera/photographer" target="_blank">https://github.com/jgonera/photographer</a>.
There is no docs or anything<br>
>>>> yet,<br>
>>>> but I prepared a simple demo patch for
MobileFrontend:<br>
>>>> <a moz-do-not-send="true" href="https://gerrit.wikimedia.org/r/#/c/126878/" target="_blank">https://gerrit.wikimedia.org/r/#/c/126878/</a>.<br>
>>>><br>
>>>> The idea is to add a new method for
Cucumber steps (snap) that takes a<br>
>>>> screenshot of the current browser state
and compares it with a<br>
>>>> screenshot<br>
>>>> taken in one of the previous test runs.
To update screenshots that are<br>
>>>> used<br>
>>>> as a reference you run tests with env
var PHOTOGRAPHER=update. If newly<br>
>>>> taken screenshot differs by too many
pixels from an old one, the test<br>
>>>> will<br>
>>>> fail.<br>
>>>><br>
>>>> It's still only an early prototype, but
I'd appreciate any comments<br>
>>>> about<br>
>>>> this idea.<br>
>>>><br>
>>>> --<br>
>>>> Juliusz<br>
>>>><br>
>>>>
_______________________________________________<br>
>>>> Mobile-l mailing list<br>
>>>> <a moz-do-not-send="true" href="mailto:Mobile-l@lists.wikimedia.org">Mobile-l@lists.wikimedia.org</a><br>
>>>> <a moz-do-not-send="true" href="https://lists.wikimedia.org/mailman/listinfo/mobile-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/mobile-l</a><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> QA mailing list<br>
>> <a moz-do-not-send="true" href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>
>> <a moz-do-not-send="true" href="https://lists.wikimedia.org/mailman/listinfo/qa" target="_blank">https://lists.wikimedia.org/mailman/listinfo/qa</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> QA mailing list<br>
> <a moz-do-not-send="true" href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>
> <a moz-do-not-send="true" href="https://lists.wikimedia.org/mailman/listinfo/qa" target="_blank">https://lists.wikimedia.org/mailman/listinfo/qa</a><br>
><br>
<br>
_______________________________________________<br>
QA mailing list<br>
<a moz-do-not-send="true" href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>
<a moz-do-not-send="true" href="https://lists.wikimedia.org/mailman/listinfo/qa" target="_blank">https://lists.wikimedia.org/mailman/listinfo/qa</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
QA mailing list
<a class="moz-txt-link-abbreviated" href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a>
<a class="moz-txt-link-freetext" href="https://lists.wikimedia.org/mailman/listinfo/qa">https://lists.wikimedia.org/mailman/listinfo/qa</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>QA mailing list<br><a href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>https://lists.wikimedia.org/mailman/listinfo/qa<br></blockquote></div><br></div></body></html>