[QA] Test failures -> Trello: Introducing Sellello

[Chris McMahon]

On Thu, May 22, 2014 at 9:08 AM, Juliusz Gonera <jgonera at wikimedia.org>wrote:

> On 05/22/2014 02:57 PM, Jon Robson wrote:
>
>> When a browser test has been failing for a considerable time e.g. for
>> the last 3 builds it will post a card to Trello.
>
>
Once a long long time ago(*) I wrote a test harness for Selenium tests from
scratch.  My framework would check the result of each individual test, and
upon the first failure of the test, would re-execute that test one time.
 Only if the individual test failed twice in a row would the framework
report an "official" failure and run the next test in the build.  I think I
first heard about this because Google did it.

I think what Jon has here is a per-build check, not per-test.  If we keep
going this way, it might be interesting to create a per-test check for
pass/fail/retry/report workflow.  Our new "Automation Engineer" starts work
May 27, this could be a good project for him.

In the meantime, if Trello is the official place of record for browser test
build results, we might want to run builds more often.  Going 1.5 days with
no failure report because of checking in the wrong PageObject would not be
good.
https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox/577/


> No card will be
>> posted on the 4th or 5th time the tests fail.
>>
>
> What about 6th and 7th and so on?
>
> --
> Juliusz
>


(*) when dinosaurs roamed the earth, before Jenkins, when CruiseControl was
the standard CI, when Selenium only injected javascript into an iframe,
when WebDriver was just an insane scheme in the fevered brain of Simon
Stewart, about 2006...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/qa/attachments/20140522/4a8e4aa3/attachment.html>