[QA] No one cares about browser tests [Re: [WikimediaMobile] MobileFrontend QA job]

[Dan Duvall]

On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson <jdlrobson at gmail.com> wrote:
> It really saddens me how very few engineers seem to care about browser
> tests. Our browser tests are failing all over the place. I just saw this bug
> [1] which has been sitting around for ages and denying us green tests in
> Echo one of our most important features.
>
> How can we change this anti-pattern?

That's exactly what I'd like to explore with you and other like minds.

> Dan Duval, would it make sense to do a survey as you did with Vagrant to
> understand how our developers think of these? Such as who owns them... who
> is responsible for a test failing... who writes them... who doesn't
> understand them.. why they don't understand them etc...?

Great idea! I suspect that the number of false positives in a given
repo's test suite is inversely related to the number of developers on
the team actually writing tests, and the affordance by managers to do
so. If you're not regularly writing tests, you're probably not going
to feel comfortable troubleshooting and refactoring someone else's. If
TDD isn't factored in to your team's velocity, you may feel like the
investment in writing tests (or learning to write them) isn't worth it
or comes at the risk of missing deadlines.

A survey could definitely help us to verify (or disprove) these relationships.

Some other questions I can think of:

 - How valuable are unit tests to the health/quality of a software project?
 - How valuable are browser tests to the health/quality of a software project?
 - How much experience do you have with TDD?
 - Would you like more time to learn or practice TDD?
 - How often do you write tests when developing a new feature?
   - What kinds of test? (% of unit test vs. browser test)
 - How often do you write tests to verify a bugfix?
   - What kinds of test? (% of unit test vs. browser test)
 - When would you typically write a unit test? (before implementation,
after implementation, when stuff breaks)
 - When would you typically write a browser test? (during conception,
before implementation, after implementation, when stuff breaks)
 - What are the largest barriers to writing/running unit tests? (test
framework, documentation/examples, execution time, CI, structure of my
code, structure of code I depend on)
 - What are the largest barriers to writing/running browser tests?
(test framework, documentation/examples, execution time, CI)
 - What are the largest barriers to debugging test failure? (test
framework, confusing errors/stack traces, documentation/examples,
debugging tools)

I'll create a Phab task to track it. :)

-- 
Dan Duvall
Automation Engineer
Wikimedia Foundation