[QA] Test failures -> Trello: Introducing Sellello

[Jon Robson]

> I think what Jon has here is a per-build check, not per-test.  If we keep
> going this way, it might be interesting to create a per-test check for
> pass/fail/retry/report workflow.  Our new "Automation Engineer" starts work
> May 27, this could be a good project for him.
Correct. This could be a logical next step but I'd like to see the
tests becoming more reliable first. I also worry if one step breaks
and 50 tests fail that's a lot of spam. I thus think a per build test
could be enough (although ideally it would be good to trigger it on a
single build and reduce flakiness)
>
> In the meantime, if Trello is the official place of record for browser test
> build results, we might want to run builds more often.  Going 1.5 days with
> no failure report because of checking in the wrong PageObject would not be
> good.
> https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox/577/

Agreed.
>
>>>
>>> No card will be
>>> posted on the 4th or 5th time the tests fail.
>>
>>
>> What about 6th and 7th and so on?
>>
>> --
4th, 5th, 6th etc...
Basically it will only make one card until you fix the problem.

Basically the logic is
1) Is the current build failing?
2) What was the build  it last passed on and was this more than or
equal to 3 builds ago?
3) Have I made a card based on that build

>> Juliusz
>
>
>
> (*) when dinosaurs roamed the earth, before Jenkins, when CruiseControl was
> the standard CI, when Selenium only injected javascript into an iframe, when
> WebDriver was just an insane scheme in the fevered brain of Simon Stewart,
> about 2006...
>
> _______________________________________________
> QA mailing list
> QA at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/qa
>