Thanks for the point-by-point notices!
On Tue, Apr 10, 2012 at 4:17 PM, Markus Glaser glaser@hallowelt.biz wrote:
- Testing complex interaction patterns with several page reloads (e.g.
maybe producing edit conflicts)
Yes, and also in environments where interactions among many extensions are not well understood.
- Testing cross-browser compatibility (e.g. using the screenshot feature)
IE should really get more attention than it does. I've found IE-only problems in a number of places.
- It's easy to record tests using the Selenium IDE. So basically anyone
could write tests, esp. for some less used extensions
Actually, creating tests with the IDE is a really bad idea. Such tests are almost always brittle and impossible to maintain. The IDE is occasionally useful for hints about how to identify tricky page elements, but really should not be used to generate whole tests.
- This also might be useful when filing bugs (make them reproducible)
Definitely.
While I see the case for new approach based on Ruby and Watir, I still think, some of the experiences we made back then could be useful in the design of the new testing environment. Here are some points of what we have so far:
- a (rudimentary) set of methods that generalize login, page calls, page
preparation with wikitext etc.
Definitely. Much of this sort of thing is implicit and automatic when using the Page Object design pattern. While it is possible to implement Page Objects in any language (Mozilla has done a good job of it in Python), Ruby is the only webdriver-supporting language with a common, documented, well-supported implementation of the pattern.
- a configuration system to run the tests against different wikis, and
also a test suite layout that helps to select a subset of tests.
This already exists using Rspec and rake.
- a command-line test runner
rake controls running Rspec tests, and Jenkins has institutional support for rake integration
* a way of reconfiguring the wiki under test so that we can standartize
some settings, such as language (which is an issue in some cases when testing UI). This also provides means to switch the wiki database to a test db and test images folder.
I'm not sure about this. Browser tests should be about controlling browsers, not databases, not files on the filesystem. I want users to be able to download the test suite and have it Just Work, whether they are on Linux, OSX, or Windows. Beta Labs wikis should help here.
- some ideas about the design of tests, suites and individual assertions
I would be happy to contribute and take some of this to the next level within the new environment :)
Thanks Markus, there is still much to be done, but I'm hoping to have something useful very soon.
-Chris
Cheers, Markus (mglaser)