<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Sure, but that doesn't blend with
current Selenium tests, does it?<br>
Also, I'm not sure if tests should pass on significant difference
in screenshots. I agree that getting this right to avoid false
alarms is tricky, but it would be useful. Otherwise, I'm afraid
nobody will pay attention.<br>
<br>
<br>
<br>
On 05/08/2014 11:57 AM, Krinkle wrote:<br>
</div>
<blockquote
cite="mid:29C63E41-05AD-4E21-921C-E9154B7BDD47@gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Could use something like pdiff:
<div><span class="Apple-tab-span" style="white-space:pre"> </span><a
moz-do-not-send="true"
href="https://bugzilla.wikimedia.org/show_bug.cgi?id=62633">https://bugzilla.wikimedia.org/show_bug.cgi?id=62633</a></div>
<div><br>
</div>
<div>Would compare the screenshot during the 'test' pipeline,
against the screenshot from the last post-merge for the same
branch.</div>
<div>Wouldn't and shouldn't give you test passing / test failing.
It would instead report to Gerrit with how much (if) there is
different (percentage/ratio or something), and you can then
follow the link to look at it yourself. Especially if you didn't
intend for visual changes.</div>
<div><br>
</div>
<div>Been using it in some smaller projects and has been very
helpful and time saving.</div>
<div><br>
<div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;">— Krinkle</div>
</div>
<br>
<div>
<div>On 6 May 2014, at 22:43, Juliusz Gonera <<a
moz-do-not-send="true" href="mailto:jgonera@wikimedia.org">jgonera@wikimedia.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
<div text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Agreed, but it would be good
for this mechanism to be as decoupled from the tests
themselves as possible. This way it can be reused and
improved by other people (outside of WMF).<br>
<br>
<br>
On 04/22/2014 08:58 AM, Chris McMahon wrote:<br>
</div>
<blockquote
cite="mid:CAJohBHSt_9Rc9PwdY3wP+mGNLY1w62sqEoHa-dgmPOXT7_YhGA@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div>Yes, but I think it might be possible to share
the underlying mechanism of taking screen shots
among all the projects that want to do that, for
whatever purposes they have. </div>
<div><br>
</div>
<div> -Chris </div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Tue, Apr 22, 2014 at 8:53
AM, Jon Robson <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:jrobson@wikimedia.org"
target="_blank">jrobson@wikimedia.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">This
wouldn't just be when a test fails. It would be
used to detect<br>
regressions in appearance at given steps....<br>
e.g. Given I go to the login page When I trigger
an error Then I see<br>
an error message And the login page looks exactly
like it was a week<br>
ago.<br>
<div class="HOEnZb">
<div class="h5"><br>
<br>
On Tue, Apr 22, 2014 at 7:20 AM, Chris McMahon
<<a moz-do-not-send="true"
href="mailto:cmcmahon@wikimedia.org">cmcmahon@wikimedia.org</a>>
wrote:<br>
><br>
><br>
> Language people also working on screen
shots:<br>
> <a moz-do-not-send="true"
href="https://gerrit.wikimedia.org/r/#/c/127899/"
target="_blank">https://gerrit.wikimedia.org/r/#/c/127899/</a><br>
><br>
><br>
> On Mon, Apr 21, 2014 at 2:25 PM, Juliusz
Gonera <<a moz-do-not-send="true"
href="mailto:jgonera@wikimedia.org">jgonera@wikimedia.org</a>><br>
> wrote:<br>
>><br>
>> Nope, it's an early prototype. It
basically only makes an updated Cucumber<br>
>> test fail if new screenshot differs
from reference screenshot by more than<br>
>> X%. Adding visual comparison of two
screenshots would be the next step and<br>
>> would not be too hard. I might work
on it one evening this week.<br>
>><br>
>><br>
>><br>
>> On 04/21/2014 11:42 AM, Tomasz Finc
wrote:<br>
>>><br>
>>> Great. Eager to see us test with
this.<br>
>>><br>
>>> Do we have any of its early
screenshot comparisons up?<br>
>>><br>
>>> --tomasz<br>
>>><br>
>>> On Fri, Apr 18, 2014 at 4:33 PM,
Juliusz Gonera <<a moz-do-not-send="true"
href="mailto:jgonera@wikimedia.org">jgonera@wikimedia.org</a>><br>
>>> wrote:<br>
>>>><br>
>>>> I conducted a little bit of
research on visual regression testing,<br>
>>>> mainly<br>
>>>> for the mobile team for now,
but it could be easily reused for other<br>
>>>> teams.<br>
>>>> I had a look at three
existing solutions that seem somewhat popular
and<br>
>>>> are<br>
>>>> actively developed:<br>
>>>><br>
>>>> * Wraith (<a
moz-do-not-send="true"
href="https://github.com/BBC-News/wraith"
target="_blank">https://github.com/BBC-News/wraith</a>)<br>
>>>> * PhantomCSS (<a
moz-do-not-send="true"
href="https://github.com/Huddle/PhantomCSS"
target="_blank">https://github.com/Huddle/PhantomCSS</a>)<br>
>>>> * Huxley (<a
moz-do-not-send="true"
href="https://github.com/facebook/huxley"
target="_blank">https://github.com/facebook/huxley</a>)<br>
>>>><br>
>>>> They all have their own pros
and cons, but in my opinion they share one<br>
>>>> important disadvantage: they
can't be easily integrated with our current<br>
>>>> browser testing setup. For
all the aforementioned tools we would have to<br>
>>>> create from scratch a
completely separate set of tests just for
visual<br>
>>>> regression testing instead of
extending our existing browser tests.<br>
>>>><br>
>>>> I spent a few hours in my
spare time tinkering with an alternative idea<br>
>>>> which would enable us to add
visual regression testing to our<br>
>>>> Cucumber/Watir<br>
>>>> tests. The result is a small
prototype available at<br>
>>>> <a moz-do-not-send="true"
href="https://github.com/jgonera/photographer"
target="_blank">https://github.com/jgonera/photographer</a>.
There is no docs or anything<br>
>>>> yet,<br>
>>>> but I prepared a simple demo
patch for MobileFrontend:<br>
>>>> <a moz-do-not-send="true"
href="https://gerrit.wikimedia.org/r/#/c/126878/"
target="_blank">https://gerrit.wikimedia.org/r/#/c/126878/</a>.<br>
>>>><br>
>>>> The idea is to add a new
method for Cucumber steps (snap) that takes a<br>
>>>> screenshot of the current
browser state and compares it with a<br>
>>>> screenshot<br>
>>>> taken in one of the previous
test runs. To update screenshots that are<br>
>>>> used<br>
>>>> as a reference you run tests
with env var PHOTOGRAPHER=update. If newly<br>
>>>> taken screenshot differs by
too many pixels from an old one, the test<br>
>>>> will<br>
>>>> fail.<br>
>>>><br>
>>>> It's still only an early
prototype, but I'd appreciate any comments<br>
>>>> about<br>
>>>> this idea.<br>
>>>><br>
>>>> --<br>
>>>> Juliusz<br>
>>>><br>
>>>>
_______________________________________________<br>
>>>> Mobile-l mailing list<br>
>>>> <a moz-do-not-send="true"
href="mailto:Mobile-l@lists.wikimedia.org">Mobile-l@lists.wikimedia.org</a><br>
>>>> <a moz-do-not-send="true"
href="https://lists.wikimedia.org/mailman/listinfo/mobile-l"
target="_blank">https://lists.wikimedia.org/mailman/listinfo/mobile-l</a><br>
>><br>
>><br>
>><br>
>>
_______________________________________________<br>
>> QA mailing list<br>
>> <a moz-do-not-send="true"
href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>
>> <a moz-do-not-send="true"
href="https://lists.wikimedia.org/mailman/listinfo/qa"
target="_blank">https://lists.wikimedia.org/mailman/listinfo/qa</a><br>
><br>
><br>
><br>
>
_______________________________________________<br>
> QA mailing list<br>
> <a moz-do-not-send="true"
href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>
> <a moz-do-not-send="true"
href="https://lists.wikimedia.org/mailman/listinfo/qa"
target="_blank">https://lists.wikimedia.org/mailman/listinfo/qa</a><br>
><br>
<br>
_______________________________________________<br>
QA mailing list<br>
<a moz-do-not-send="true"
href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>
<a moz-do-not-send="true"
href="https://lists.wikimedia.org/mailman/listinfo/qa"
target="_blank">https://lists.wikimedia.org/mailman/listinfo/qa</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
QA mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="https://lists.wikimedia.org/mailman/listinfo/qa">https://lists.wikimedia.org/mailman/listinfo/qa</a>
</pre>
</blockquote>
<br>
</div>
_______________________________________________<br>
QA mailing list<br>
<a moz-do-not-send="true"
href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a><br>
<a class="moz-txt-link-freetext" href="https://lists.wikimedia.org/mailman/listinfo/qa">https://lists.wikimedia.org/mailman/listinfo/qa</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
QA mailing list
<a class="moz-txt-link-abbreviated" href="mailto:QA@lists.wikimedia.org">QA@lists.wikimedia.org</a>
<a class="moz-txt-link-freetext" href="https://lists.wikimedia.org/mailman/listinfo/qa">https://lists.wikimedia.org/mailman/listinfo/qa</a>
</pre>
</blockquote>
<br>
</body>
</html>