It really saddens me how very few engineers seem to care about browser
tests. Our browser tests are failing all over the place. I just saw this
bug [1] which has been sitting around for ages and denying us green tests
in Echo one of our most important features.
How can we change this anti-pattern?
Dan Duval, would it make sense to do a survey as you did with Vagrant to
understand how our developers think of these? Such as who owns them... who
is responsible for a test failing... who writes them... who doesn't
understand them.. why they don't understand them etc...?
[1]
On Thu, Mar 26, 2015 at 5:54 PM, Dan Duvall <dduvall(a)wikimedia.org> wrote:
On Thu, Mar 26, 2015 at 3:38 PM, Jon Robson
<jdlrobson(a)gmail.com> wrote:
The lack of replies from any mobile team members
seems to suggest I'm the
only one watching this. In that case I'd rather we split up the big
MobileFrontend job into various jobs based on certain features. How can
we
get more people caring about browser tests?
Naming and shaming?
We have a couple projects in the works that will hopefully help.[1][2]
They don't go so far as to name and shame (we'll leave that up to you
:), but they should promote more ownership over browser tests, better
communication of failure, and analysis of failure over time and by
test function (feature vs smoke vs regression).
If many of these browser tests are serving as regression/smoke tests,
it might be worthwhile to not only separate them out, but to remove
some of the layers of abstraction to make tests more maintainable. I
often try to think about tests in terms of a value-to-debt ratio (i.e.
"How likely is it that I'll have to fix or refactor this test down the
road and is it worth it?") and, while I find quite a bit of value in
Cucumber for the sake of acceptance(-esque) testing (it keeps me
mindful of the UX),[3] it introduces quite a bit of debt in its layers
of abstraction and indirection (it's always a red flag when you have
to grep to find your test implementation). Even its creators believe
the value of Cucumber lies in its collaboration/planning capacities,
not in its function as a test runner.[4]
It's entirely possible, as of the 1.0.0 release, to use MW-Selenium
without Cucumber, so perhaps we could experiment with simple RSpec
examples for these types of tests.
Tooling aside, there are broader discussions that need to take place
among those that are practicing or have practiced TDD/BDD in the
organization and how we might (or might not) promote theses practices
among others. I'll be trying to form such a group next quarter (will
it be a 'guild'?, a 'league'?, no need for tough decisions just
yet[5]), so let's maintain a dialogue on that front if you're
interested and willing.
[1]
https://phabricator.wikimedia.org/T88706
[2]
https://phabricator.wikimedia.org/T89375
[3]
https://gerrit.wikimedia.org/r/#/c/196492/10/features/role_settings.feature
[4]
https://cukes.info/blog/2014/03/03/the-worlds-most-misunderstood-collaborat…
[5]
https://www.youtube.com/watch?v=6KSiyaqnZYs
Practically, the current big job [1] we have has
far too many false
positives and it is just noise to me. I was paying attention to the smoke
tests [2] but I was told that was an upstream bug and haven't been
watching
it since. Any idea why that has been failing
recently? Nothing has broken
here...
[1]
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFro…
[2]
https://integration.wikimedia.org/ci/view/Mobile/job/browsertests-MobileFro…
Have you tried to reproduce these failures by executing the tests
locally, against either Beta or your local MW? I'll be in the office
tomorrow if you want advice on how to best go about it.
Dan
--
Dan Duvall
Automation Engineer
Wikimedia Foundation