On 05/02/2010 01:20 AM, Markus Glaser wrote:
at the Wkimedia Developers' Workshop, I introduced a Selenium testing framework for
MediaWiki. Since it has now been promoted to maintenance/tests, I have provided some
initial information it on http://www.mediawiki.org/wiki/SeleniumFramework
. I would be
very happy about comments and ideas for further improvement. Also, if you intend to use
the framework for your tests, please let me know. I will be happy to assist.
It's a good leap in the right direction, a few comments:
I spent about half an hour trying to get PagedDiffHandler tests to run,
(though some uploads still seem to "fail" by uploading successfully):
* Please distribute all the images needed with the extension, I followed
the link in SOURCES.txt, and found that the link on that page was broken
(though it wasn't hard to find a fixed link).
* Don't hardcode "\\" as the directory separator - windows is not here!
* Use parser tests instead of selenium where appropriate (this will save
you "lots" of typing and a bit of test execution time)
* Make the tests pass by default :p.
Comments about the framework itself:
* Don't hardcode the firefox path - selenium's guess is better than yours.
* Should always shutdown selenium properly (I was often left with
myriads of firefoxes)
* !Always run against a clean database! (it is unrealistic to expect
people to write perfect tests, and particularly while developing the
tests having an automatically clean database is essential).
* Get configuration from the main LocalSettings.php and use a mechanism
like parser tests for registering selenium test files. (making it as
easy as possible to run tests is a good idea).
<waffle> How to deal with configuration (and by extension databases) for
selenium tests... It is a tricky question, and to solve it properly
requires a bit of effort. I think you need to send a selenium token with
each request (like you suggested, though it'd be nicer to use a cookie
rather than a url parameter in my opinion).
Either you could do a log in and get a session id, or you could simply
update the cookie on every request with:
(selenium_test_name, timestamp, start_of_test,
HASH(selenium_test_name, timestamp, shared_secret))
Providing mediawiki is in selenium mode, and the timestamp is recent,
the selenium_test_name matches a registered selenium test and the hash
is correct, then we can be reasonably sure this is a valid selenium
request (I assume no-one steals the secret, much as you assume no-one
reads wikisysop's password out of the file - besides you can always
update it, and it's only a problem if selenium mode is enabled).
If timestamp == start_of_test then you can clean the test databases/
upload directories (or do that on "log in"). You can then also load any
test-specific configuration from the selenium_test_name test on every
request (and change cache-prefixes and upload directories to avoid
If you allow for one database (or pseudo database with table prefixes,
like parser test) per selenium_test_name then you can run different
tests in parallel; otherwise you could generate a new prefix based on
the start_of_test and the selenium_test_name - but you'd quickly need a
way to delete the databases completely once you'd finished.</waffle>
Thanks for your great work so far!