[QA] No one cares about browser tests [Re: [WikimediaMobile] MobileFrontend QA job]

[Jon Robson]

It really saddens me how very few engineers seem to care about browser
tests. Our browser tests are failing all over the place. I just saw this
bug [1] which has been sitting around for ages and denying us green tests
in Echo one of our most important features.

How can we change this anti-pattern?

Dan Duval, would it make sense to do a survey as you did with Vagrant to
understand how our developers think of these? Such as who owns them... who
is responsible for a test failing... who writes them... who doesn't
understand them.. why they don't understand them etc...?

[1] https://phabricator.wikimedia.org/T72298

On Thu, Mar 26, 2015 at 5:54 PM, Dan Duvall <dduvall at wikimedia.org> wrote:

> On Thu, Mar 26, 2015 at 3:38 PM, Jon Robson <jdlrobson at gmail.com> wrote:
> > The lack of replies from any mobile team members seems to suggest I'm the
> > only one watching this. In that case I'd rather we split up the big
> > MobileFrontend job into various jobs based on certain features. How can
> we
> > get more people caring about browser tests? Naming and shaming?
>
> We have a couple projects in the works that will hopefully help.[1][2]
> They don't go so far as to name and shame (we'll leave that up to you
> :), but they should promote more ownership over browser tests, better
> communication of failure, and analysis of failure over time and by
> test function (feature vs smoke vs regression).
>
> If many of these browser tests are serving as regression/smoke tests,
> it might be worthwhile to not only separate them out, but to remove
> some of the layers of abstraction to make tests more maintainable. I
> often try to think about tests in terms of a value-to-debt ratio (i.e.
> "How likely is it that I'll have to fix or refactor this test down the
> road and is it worth it?") and, while I find quite a bit of value in
> Cucumber for the sake of acceptance(-esque) testing (it keeps me
> mindful of the UX),[3] it introduces quite a bit of debt in its layers
> of abstraction and indirection (it's always a red flag when you have
> to grep to find your test implementation). Even its creators believe
> the value of Cucumber lies in its collaboration/planning capacities,
> not in its function as a test runner.[4]
>
> It's entirely possible, as of the 1.0.0 release, to use MW-Selenium
> without Cucumber, so perhaps we could experiment with simple RSpec
> examples for these types of tests.
>
> Tooling aside, there are broader discussions that need to take place
> among those that are practicing or have practiced TDD/BDD in the
> organization and how we might (or might not) promote theses practices
> among others. I'll be trying to form such a group next quarter (will
> it be a 'guild'?, a 'league'?, no need for tough decisions just
> yet[5]), so let's maintain a dialogue on that front if you're
> interested and willing.
>
> [1] https://phabricator.wikimedia.org/T88706
> [2] https://phabricator.wikimedia.org/T89375
> [3]
> https://gerrit.wikimedia.org/r/#/c/196492/10/features/role_settings.feature
> [4]
> https://cukes.info/blog/2014/03/03/the-worlds-most-misunderstood-collaboration-tool
> [5] https://www.youtube.com/watch?v=6KSiyaqnZYs
>
> > Practically, the current big job [1] we have has far too many false
> > positives and it is just noise to me. I was paying attention to the smoke
> > tests [2] but I was told that was an upstream bug and haven't been
> watching
> > it since. Any idea why that has been failing recently? Nothing has broken
> > here...
> >
> > [1]
> >
> https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/
> > [2]
> >
> https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/
> >
>
> Have you tried to reproduce these failures by executing the tests
> locally, against either Beta or your local MW? I'll be in the office
> tomorrow if you want advice on how to best go about it.
>
> Dan
>
> --
> Dan Duvall
> Automation Engineer
> Wikimedia Foundation
>



-- 
Jon Robson
* http://jonrobson.me.uk
* https://www.facebook.com/jonrobson
* @rakugojon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20150330/4b49ff6c/attachment.html>