[QA] No one cares about browser tests [Re: [WikimediaMobile] MobileFrontend QA job]

[Dan Duvall]

https://phabricator.wikimedia.org/T94472

On Mon, Mar 30, 2015 at 12:39 PM, Dan Duvall <dduvall at wikimedia.org> wrote:
> On Mon, Mar 30, 2015 at 10:30 AM, Jon Robson <jdlrobson at gmail.com> wrote:
>> It really saddens me how very few engineers seem to care about browser
>> tests. Our browser tests are failing all over the place. I just saw this bug
>> [1] which has been sitting around for ages and denying us green tests in
>> Echo one of our most important features.
>>
>> How can we change this anti-pattern?
>
> That's exactly what I'd like to explore with you and other like minds.
>
>> Dan Duval, would it make sense to do a survey as you did with Vagrant to
>> understand how our developers think of these? Such as who owns them... who
>> is responsible for a test failing... who writes them... who doesn't
>> understand them.. why they don't understand them etc...?
>
> Great idea! I suspect that the number of false positives in a given
> repo's test suite is inversely related to the number of developers on
> the team actually writing tests, and the affordance by managers to do
> so. If you're not regularly writing tests, you're probably not going
> to feel comfortable troubleshooting and refactoring someone else's. If
> TDD isn't factored in to your team's velocity, you may feel like the
> investment in writing tests (or learning to write them) isn't worth it
> or comes at the risk of missing deadlines.
>
> A survey could definitely help us to verify (or disprove) these relationships.
>
> Some other questions I can think of:
>
>  - How valuable are unit tests to the health/quality of a software project?
>  - How valuable are browser tests to the health/quality of a software project?
>  - How much experience do you have with TDD?
>  - Would you like more time to learn or practice TDD?
>  - How often do you write tests when developing a new feature?
>    - What kinds of test? (% of unit test vs. browser test)
>  - How often do you write tests to verify a bugfix?
>    - What kinds of test? (% of unit test vs. browser test)
>  - When would you typically write a unit test? (before implementation,
> after implementation, when stuff breaks)
>  - When would you typically write a browser test? (during conception,
> before implementation, after implementation, when stuff breaks)
>  - What are the largest barriers to writing/running unit tests? (test
> framework, documentation/examples, execution time, CI, structure of my
> code, structure of code I depend on)
>  - What are the largest barriers to writing/running browser tests?
> (test framework, documentation/examples, execution time, CI)
>  - What are the largest barriers to debugging test failure? (test
> framework, confusing errors/stack traces, documentation/examples,
> debugging tools)
>
> I'll create a Phab task to track it. :)
>
> --
> Dan Duvall
> Automation Engineer
> Wikimedia Foundation



-- 
Dan Duvall
Automation Engineer
Wikimedia Foundation