[QA] [WikimediaMobile] MobileFrontend QA job

[Dan Duvall]

On Thu, Mar 26, 2015 at 3:38 PM, Jon Robson <jdlrobson at gmail.com> wrote:
> The lack of replies from any mobile team members seems to suggest I'm the
> only one watching this. In that case I'd rather we split up the big
> MobileFrontend job into various jobs based on certain features. How can we
> get more people caring about browser tests? Naming and shaming?

We have a couple projects in the works that will hopefully help.[1][2]
They don't go so far as to name and shame (we'll leave that up to you
:), but they should promote more ownership over browser tests, better
communication of failure, and analysis of failure over time and by
test function (feature vs smoke vs regression).

If many of these browser tests are serving as regression/smoke tests,
it might be worthwhile to not only separate them out, but to remove
some of the layers of abstraction to make tests more maintainable. I
often try to think about tests in terms of a value-to-debt ratio (i.e.
"How likely is it that I'll have to fix or refactor this test down the
road and is it worth it?") and, while I find quite a bit of value in
Cucumber for the sake of acceptance(-esque) testing (it keeps me
mindful of the UX),[3] it introduces quite a bit of debt in its layers
of abstraction and indirection (it's always a red flag when you have
to grep to find your test implementation). Even its creators believe
the value of Cucumber lies in its collaboration/planning capacities,
not in its function as a test runner.[4]

It's entirely possible, as of the 1.0.0 release, to use MW-Selenium
without Cucumber, so perhaps we could experiment with simple RSpec
examples for these types of tests.

Tooling aside, there are broader discussions that need to take place
among those that are practicing or have practiced TDD/BDD in the
organization and how we might (or might not) promote theses practices
among others. I'll be trying to form such a group next quarter (will
it be a 'guild'?, a 'league'?, no need for tough decisions just
yet[5]), so let's maintain a dialogue on that front if you're
interested and willing.

[1] https://phabricator.wikimedia.org/T88706
[2] https://phabricator.wikimedia.org/T89375
[3] https://gerrit.wikimedia.org/r/#/c/196492/10/features/role_settings.feature
[4] https://cukes.info/blog/2014/03/03/the-worlds-most-misunderstood-collaboration-tool
[5] https://www.youtube.com/watch?v=6KSiyaqnZYs

> Practically, the current big job [1] we have has far too many false
> positives and it is just noise to me. I was paying attention to the smoke
> tests [2] but I was told that was an upstream bug and haven't been watching
> it since. Any idea why that has been failing recently? Nothing has broken
> here...
>
> [1]
> https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/
> [2]
> https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/
>

Have you tried to reproduce these failures by executing the tests
locally, against either Beta or your local MW? I'll be in the office
tomorrow if you want advice on how to best go about it.

Dan

-- 
Dan Duvall
Automation Engineer
Wikimedia Foundation