On 16/04/12 16:45, Chris McMahon wrote:
Cucumber adds a layer of abstraction I think is unnecessary-- these tests are to be read by developers, many of whom will not be expert. Rspec is a nice alternative to xUnit-style assertions, and the standard among Ruby developers.
Which seem to be the only ones that will be able to write our tests.
You can talk about how "there's a standard way" or "it's documented", but that doesn't help if they aren't easy to develop. For example X.509 is all standard you want, really flexible, but you wouldn't want to manually touch it :) Moreover, making tests is not generally fun to do, so you want the barrier to be as low as possible.
My idea right now is that maintaining the Selenium test suite as run by Jenkins would be primarily a QA activity, with contributions from any other interested parties in the greater community or among the Mediawiki/Wikipedia dev/ops community. Contributions from developers would be welcome but not required.
Not that I have anything against making tests by opening bugs, and letting the QA team (who forms that?) struggle to make them work. It's simple to just be lazy and let you test it manually, with a monkey farm, gems or a street rat. But I don't think that's the way we should go. The developers should be able to maintain them, even if they're not daily involved with it. Not only for being able to keep them the day you stop doing it, but also for detecting why they fail and maintaining them when changing the expected output.
It wouldn't be fun to held a discussion in gerrit about why a bunch of tests suddenly fail after merging a big feature, wondering what it could have been trying to test.