[QA] Signal vs noise

[Željko Filipin]

James and I are having an interesting discussion at Gerrit[0], but I think
it is not the right place for the discussion, so I am moving it here.

Story #1

During Wikimania, Vikas and I worked on a new feature for VisualEditor
browser screenshots. The feature broke all VisualEditor browser tests, but
we did not notice.

The problem is that nobody noticed. Or at least, nobody complained. Our
browser tests[1] are usually so broken, that nobody noticed when they got
even more broken.

Story #2

Looks like all browser tests for all browsers except Firefox and Chrome are
broken since mediawiki_selenium 1.0. (We are at 1.6 now.) Again, nobody
noticed. Or at least, nobody complained. Until I have noticed it[2].

Epilogue

To fix the problem, I have started cleaning up jobs that are broken for the
last month[3]. Why? I think it is better if a repository has only one (1!)
browser test that is consistently green than more tests that are failing
all the time.

Why? When that one tests breaks, it signals a problem. When all the tests
are broken all the time, there is no signal. Only noise.

Željko
--
0: https://gerrit.wikimedia.org/r/#/c/247612/
1: https://integration.wikimedia.org/ci/view/BrowserTests/view/-All/
2: https://phabricator.wikimedia.org/T114362
3: https://phabricator.wikimedia.org/T94150

On Mon, Nov 2, 2015 at 4:22 PM, Jforrester (Code Review) <
gerrit at wikimedia.org> wrote:

> Jforrester has posted comments on this change.
>
> Change subject: Revert "Delete failing VisualEditor browsertests Jenkins
> jobs"
> ......................................................................
>
>
> Patch Set 2:
>
> > I disagree that VisualEditor browser tests were fine before we deleted
> the jobs.
>
> No, but they were fine (some working, some not, in the slow process of
> being fixed despite being foisted upon the team) before you broke them.
>
> > I have asked on several places (qa mailing list, phabricator, gerrit)
> before deleting the jobs and got no reply, so I have assumed nobody cared.
>
> None of these are appropriate venues for synchronous or semi-synchronous
> communication. IRC exists. E-mail exists. Silence is not consent.
>
> > Reverting this change would show that all Internet Explorer and Safari
> jobs are broken because of T114362.
>
> OK, so… why is that task not, for instance, a high-priority
> tell-all-engineers task? Why is it that the first I hear that the CI
> infrastructure is totally broken for the most important targets a comment
> in this commit? We already trivially test everything in Chrome/Firefox as
> part of normal development; the main value of browser tests are IE, Safari
> and particularly iOS Safari…
>
> > As a first step in recreating the jobs, I will recreate one Firefox job
> and see what needs to be done to make it green.
>
> Thank you.
>
> --
> To view, visit https://gerrit.wikimedia.org/r/247612
> To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
>
> Gerrit-MessageType: comment
> Gerrit-Change-Id: Ia68624f3672f634012749fb893577a5a0e626bb8
> Gerrit-PatchSet: 2
> Gerrit-Project: integration/config
> Gerrit-Branch: master
> Gerrit-Owner: Jforrester <jforrester at wikimedia.org>
> Gerrit-Reviewer: Amire80 <amir.aharoni at mail.huji.ac.il>
> Gerrit-Reviewer: Dduvall <dduvall at wikimedia.org>
> Gerrit-Reviewer: Jforrester <jforrester at wikimedia.org>
> Gerrit-Reviewer: Vikassy <vikasyaligar.it at gmail.com>
> Gerrit-Reviewer: Vikasyaligar <vikasyaligar at gmail.com>
> Gerrit-Reviewer: Zfilipin <zfilipin at wikimedia.org>
> Gerrit-Reviewer: jenkins-bot <>
> Gerrit-HasComments: No
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20151106/be89eba7/attachment.html>