On 05/02/2010 01:20 AM, Markus Glaser wrote:
Hi everybody,
at the Wkimedia Developers' Workshop, I introduced a Selenium testing framework for MediaWiki. Since it has now been promoted to maintenance/tests, I have provided some initial information it on http://www.mediawiki.org/wiki/SeleniumFramework . I would be very happy about comments and ideas for further improvement. Also, if you intend to use the framework for your tests, please let me know. I will be happy to assist.
Regards, Markus Glaser
It's a good leap in the right direction, a few comments:
I spent about half an hour trying to get PagedDiffHandler tests to run, (though some uploads still seem to "fail" by uploading successfully):
* Please distribute all the images needed with the extension, I followed the link in SOURCES.txt, and found that the link on that page was broken (though it wasn't hard to find a fixed link). * Don't hardcode "\" as the directory separator - windows is not here! * Use parser tests instead of selenium where appropriate (this will save you "lots" of typing and a bit of test execution time) * Make the tests pass by default :p.
Comments about the framework itself:
* Don't hardcode the firefox path - selenium's guess is better than yours. * Should always shutdown selenium properly (I was often left with myriads of firefoxes) * !Always run against a clean database! (it is unrealistic to expect people to write perfect tests, and particularly while developing the tests having an automatically clean database is essential). * Get configuration from the main LocalSettings.php and use a mechanism like parser tests for registering selenium test files. (making it as easy as possible to run tests is a good idea).
<waffle> How to deal with configuration (and by extension databases) for selenium tests... It is a tricky question, and to solve it properly requires a bit of effort. I think you need to send a selenium token with each request (like you suggested, though it'd be nicer to use a cookie rather than a url parameter in my opinion).
Either you could do a log in and get a session id, or you could simply update the cookie on every request with:
(selenium_test_name, timestamp, start_of_test, HASH(selenium_test_name, timestamp, shared_secret))
Providing mediawiki is in selenium mode, and the timestamp is recent, the selenium_test_name matches a registered selenium test and the hash is correct, then we can be reasonably sure this is a valid selenium request (I assume no-one steals the secret, much as you assume no-one reads wikisysop's password out of the file - besides you can always update it, and it's only a problem if selenium mode is enabled).
If timestamp == start_of_test then you can clean the test databases/ upload directories (or do that on "log in"). You can then also load any test-specific configuration from the selenium_test_name test on every request (and change cache-prefixes and upload directories to avoid collisions).
If you allow for one database (or pseudo database with table prefixes, like parser test) per selenium_test_name then you can run different tests in parallel; otherwise you could generate a new prefix based on the start_of_test and the selenium_test_name - but you'd quickly need a way to delete the databases completely once you'd finished.</waffle>
Thanks for your great work so far!
Conrad