[QA] [reading-wmf] Recently failing browser tests

[Dan Duvall]

On Thu, Jun 4, 2015 at 11:38 AM, Gergo Tisza <gtisza at wikimedia.org> wrote:

> On Thu, Jun 4, 2015 at 11:23 AM, Jon Robson <jrobson at wikimedia.org> wrote:
>
>> Another thing I'd be interested in exploring is using JavaScript to write
>> tests since a lot more devs are comfortable with JS than Ruby but I suspect
>> this is a big undertaking?
>>
>
> This is probably not what you meant but the ratio of browser tests to unit
> tests (QUnit) in Gather seems very unhealthy to me. QUnit tests run in a
> headless browser are very fast so they can be run pre-merge, they are less
> flaky, fully parallel, and catching errors via unit tests is generally more
> productive as they can pinpoint which part of the code is wrong while
> browser tests usually need a fair amount of debugging. IMO if you are
> looking for low-hanging fruits then your time is better spent on writing
> more QUnit tests than doing complicated things with browser tests.
>

+2^64

In my experience, unit testing involves more upfront pain as it forces you
to constantly reexamine your implementation, but the payoff has always been
well worth it. It also requires a change in your workflow which needs to be
respected by leadership. One of my thoughts behind the testing working
group is to share techniques for good test design (and horror stories) and
how we might diffuse these practices across teams.


Re: browser test language, there has been some debate about this (both PHP
> and JS have mature gherkin-based tools, and most developers are more
> familiar with them than with Ruby), but the opinion of QA at the time was
> that a language which the browser testing community tends to be familiar
> with is better than a language the MediaWiki community tends to be familiar
> with.
>

Over the past year (and 10 days) since I've been at WMF, I've spent a good
amount of my time delving into browser tests repos, pairing, hacking on
MW-Selenium, and talking with developers about the pain points of writing
these tests. Take this with my inherent bias as a Ruby developer—but I know
of others in the org that would concur—that the obtuseness of our test
framework comes not so much from the language but from the level of
indirection that we've incurred by stacking abstraction upon abstraction.[1]

In many cases—again, I think we need to make these cases clear by
differentiating our tests and testing requirements—Cucumber (Gherkin) is
probably the biggest offender. It adds a ton of indirection, non-idiomatic
Ruby, unclear scope by abusing Ruby's `instance_eval` capabilities, and a
natural language that really only adds value in the context of true
acceptance testing and project management.[2] Down the road, if we want to
experiment with real user-story driven acceptance testing, we can
reintroduce this layer for those tests.

Since MW-Selenium 1.x, the framework is a lot more modular and contains
very little coupling to Cucumber. It would be straightforward to refactor
the remaining bits out to a mw-selenium-cucumber binding library and
experiment with the core framework via regular ol' RSpec. This kind of
setup would seem to lend itself well to regression tests and smoke tests
where the test case is very targeted, natural language abstraction is
really unnecessary, and linear implementation is much easier to reason
about.

While we're at it, Watir and PageObject could probably go as well. (The
latter provides a very important encapsulation of the coupling in our tests
(DOM selectors) but it's merely a pattern and the implementation could be
simplified greatly.)

That all said, I'm definitely for experimenting with alternative
frameworks/libraries, and I don't think anyone is completely married to
Ruby. I'd just hate to throw the MW-Selenium baby out with the bathwater,
just to build another baby fraught with the same inherent fragility that
comes with tests that drive a browser.

[1] https://doc.wikimedia.org/rubygems/mediawiki-selenium/#Core_Dependencies
[2]
https://cucumber.io/blog/2014/03/03/the-worlds-most-misunderstood-collaboration-tool

-- 
Dan Duvall
Automation Engineer
Wikimedia Foundation <http://wikimediafoundation.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20150604/257c570f/attachment-0001.html>