Hi all,
So as should be known by now we're going with QUnit for writing and running unit test of JavaScript and PHPUnit for writing and running unit test for our PHP code base. One can manually run both of these locally without problems (phpunit from CLI and QUnit test from the browser at [mw]/tests/qunit/index.php [1]).
As automated continuous integration we're letting Jenkins update mediawiki, run the tests and keep track of everything (including notification to IRC).
Although it's theoretically possible to let Jenkins run QUnit tests (node.js from shell running synchronously), what we really want is throwing all javascript unit tests onto our swarm of actual browsers. For that we use TestSwarm, but it's downside is that it's not as cool as Jenkins in providing statistics and insight into the state of our code.
The jQuery Testing team found a cool solution. Hooking up TestSwarm into Jenkins.
See their instance at http://swarm.jquery.org:8080/#jenkins
Now one thing that might throw you off is the 'running build'. Jenkins is not controlling the runs, as it shouldn't. TestSwarm is controlling this through the swarm. Jenkins is merely aggregating results, which, depending on the definition of "build ready" can take a long time if one of the required browsers isn't connected to the swarm. We will probably have to find a way to make this look better and have statistics available even when not every browser is connected so that we don't have to wait for "build ready", which has a different meaning when working with multiple clients and 100s of asynchronous runs.
But overall it's very nice, I think we should take the same approach for MediaWIki. Instead of trying to make TestSwarm do something it's not designed for, let TestSwarm do what it's good at, and let Jenkins do what it is good at.
More info: * Project planning page by the jQuery Testing team: http://jquerytesting.pbworks.com/w/page/43991777/TestSwarm-Jenkins-Integrati... * Fork it on Github: https://github.com/appendto/jenkins-testswarm * The jQuery Testing team
-- Krinkle
PS: Before we can use this though there are many things to do first on integration.mediawiki.org; Such as installing TestSwarm, merging JSTesting branch into trunk, setting up and testing the TestSwarmMwFetcher.. Just wanted to get this idea out there before someone might waste time trying to do the same.
[1] Soon to be /wiki/Special:JavaScriptTest/qunit :) (see /branches/JSTesting/)