Hey all,
As of today, we automatically run our QUnit test suite[4] in MediaWiki
core from Jenkins.
Example:
*
https://gerrit.wikimedia.org/r/52177
*
https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/
*
https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console
Today I sprinted to pick up QUnit testing in Jenkins and get it
stabilised and deployed.
It is run by using PhantomJS[2] and we're using
grunt-contrib-qunit[3][4] to abstract the logic:
* starting phantomjs
* pointing it to a url
* hooking into javascript engine to register callbacks to QUnit events
* progress indicator in cli
* return the proper exit code
I won't go in detail about what PhantomJS is, but in short:
It is a "headless" WebKit browser. Meaning, it doesn't render pixels
to a screen on the server side, but it does behave like a fully valid
browser environment as if it were rendering it to a screen (CSS is
being parsed, the DOM is there, stylesheets are active, retrieving
computed styles, ajax requests can be made etc.).
For more information, see [2].
Just to point out the obvious, this doesn't catch issues specific to
certain browsers (e.g. syntax only breaking in older EcmaScript 3
engines, or code that incorrectly relies on HTML5 DOM APIs that exist
in latest WebKit but not in Internet Explorer or Firefox). Those we
will catch once we also run this on multiple operating systems and in
more than 1 browser (which is the next chapter in implementing the
continuous integration workflow[5]).
Things this will catch are basically everything else. Any runtime error
that we can't detect in static analysis but will fail no matter what
browser you're in, such as:
* misspelled identifiers or syntax errors
* issues with ResourceLoader (mw.loader)
* issues with AJAX
* any code failures that result in exceptions
* the obvious (catching failures/regressions in our QUnit tests)
Even code that doesn't have unit tests. The code execution alone
should result an uncaught exception, which we can now get our hands on
since we actually execute the javascript in a real browser. This
includes anything related to ResourceLoader, since we don't just
execute the unit tests in a browser, we load them from MediaWiki's
core/index.php entry point (Special:JavaScriptTest, to be specific).
Similar to how we have the php-checkstyle job currently, the QUnit job
is in non-voting mode. However, unlike php-checkstyle, our QUnit tests
are actually passing, but we're not letting it vote yet to see how it
behaves over the next 24 hours. If it is stable, we'll make it voting
(like phplint, jshint and phpunit are already).
So, next time you read the jenkins-job comment, look for the QUnit job.
Happy testing,
-- Krinkle
[1]
https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing
[2] PhantomJS:
http://phantomjs.org/
[3] node-phantomjs (npm wrapper with nom-install hook)
https://github.com/Obvious/phantomjs
[4] grunt-contrib-qunit
https://github.com/gruntjs/grunt-contrib-qunit
[5]
https://www.mediawiki.org/wiki/Continuous_integration/Workflow_specification