On 16/04/12 16:45, Chris McMahon wrote:
Cucumber adds a layer of abstraction I think is
unnecessary-- these tests
are to be read by developers, many of whom will not be expert. Rspec is a
nice alternative to xUnit-style assertions, and the standard among Ruby
developers.
Which seem to be the only ones that will be able to write our tests.
You can talk about how "there's a standard way" or "it's
documented",
but that doesn't help if they aren't easy to develop.
For example X.509 is all standard you want, really flexible, but you
wouldn't want to manually touch it :)
Moreover, making tests is not generally fun to do, so you want the
barrier to be as low as possible.
My idea right now is that maintaining the Selenium
test suite as run by
Jenkins would be primarily a QA activity, with contributions from any other
interested parties in the greater community or among the
Mediawiki/Wikipedia dev/ops community. Contributions from developers would
be welcome but not required.
Not that I have anything against making tests by opening bugs, and
letting the QA team (who forms that?) struggle to make them work.
It's simple to just be lazy and let you test it manually, with a monkey
farm, gems or a street rat. But I don't think that's the way we should go.
The developers should be able to maintain them, even if they're not
daily involved with it. Not only for being able to keep them the day you
stop doing it, but also for detecting why they fail and maintaining them
when changing the expected output.
It wouldn't be fun to held a discussion in gerrit about why a bunch of
tests suddenly fail after merging a big feature, wondering what it could
have been trying to test.