Hey all,
As of today, we automatically run our QUnit test suite[4] in MediaWiki core from Jenkins.
Example: * https://gerrit.wikimedia.org/r/52177 * https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/ * https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console
Today I sprinted to pick up QUnit testing in Jenkins and get it stabilised and deployed.
It is run by using PhantomJS[2] and we're using grunt-contrib-qunit[3][4] to abstract the logic:
* starting phantomjs * pointing it to a url * hooking into javascript engine to register callbacks to QUnit events * progress indicator in cli * return the proper exit code
I won't go in detail about what PhantomJS is, but in short:
It is a "headless" WebKit browser. Meaning, it doesn't render pixels to a screen on the server side, but it does behave like a fully valid browser environment as if it were rendering it to a screen (CSS is being parsed, the DOM is there, stylesheets are active, retrieving computed styles, ajax requests can be made etc.). For more information, see [2].
Just to point out the obvious, this doesn't catch issues specific to certain browsers (e.g. syntax only breaking in older EcmaScript 3 engines, or code that incorrectly relies on HTML5 DOM APIs that exist in latest WebKit but not in Internet Explorer or Firefox). Those we will catch once we also run this on multiple operating systems and in more than 1 browser (which is the next chapter in implementing the continuous integration workflow[5]).
Things this will catch are basically everything else. Any runtime error that we can't detect in static analysis but will fail no matter what browser you're in, such as:
* misspelled identifiers or syntax errors * issues with ResourceLoader (mw.loader) * issues with AJAX * any code failures that result in exceptions * the obvious (catching failures/regressions in our QUnit tests)
Even code that doesn't have unit tests. The code execution alone should result an uncaught exception, which we can now get our hands on since we actually execute the javascript in a real browser. This includes anything related to ResourceLoader, since we don't just execute the unit tests in a browser, we load them from MediaWiki's core/index.php entry point (Special:JavaScriptTest, to be specific).
Similar to how we have the php-checkstyle job currently, the QUnit job is in non-voting mode. However, unlike php-checkstyle, our QUnit tests are actually passing, but we're not letting it vote yet to see how it behaves over the next 24 hours. If it is stable, we'll make it voting (like phplint, jshint and phpunit are already).
So, next time you read the jenkins-job comment, look for the QUnit job.
Happy testing,
-- Krinkle
[1] https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing
[2] PhantomJS: http://phantomjs.org/
[3] node-phantomjs (npm wrapper with nom-install hook) https://github.com/Obvious/phantomjs
[4] grunt-contrib-qunit https://github.com/gruntjs/grunt-contrib-qunit
[5] https://www.mediawiki.org/wiki/Continuous_integration/Workflow_specification