Hey all,
As of today, we automatically run our QUnit test suite[4] in MediaWiki core from Jenkins.
Example: * https://gerrit.wikimedia.org/r/52177 * https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/ * https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console
Today I sprinted to pick up QUnit testing in Jenkins and get it stabilised and deployed.
It is run by using PhantomJS[2] and we're using grunt-contrib-qunit[3][4] to abstract the logic:
* starting phantomjs * pointing it to a url * hooking into javascript engine to register callbacks to QUnit events * progress indicator in cli * return the proper exit code
I won't go in detail about what PhantomJS is, but in short:
It is a "headless" WebKit browser. Meaning, it doesn't render pixels to a screen on the server side, but it does behave like a fully valid browser environment as if it were rendering it to a screen (CSS is being parsed, the DOM is there, stylesheets are active, retrieving computed styles, ajax requests can be made etc.). For more information, see [2].
Just to point out the obvious, this doesn't catch issues specific to certain browsers (e.g. syntax only breaking in older EcmaScript 3 engines, or code that incorrectly relies on HTML5 DOM APIs that exist in latest WebKit but not in Internet Explorer or Firefox). Those we will catch once we also run this on multiple operating systems and in more than 1 browser (which is the next chapter in implementing the continuous integration workflow[5]).
Things this will catch are basically everything else. Any runtime error that we can't detect in static analysis but will fail no matter what browser you're in, such as:
* misspelled identifiers or syntax errors * issues with ResourceLoader (mw.loader) * issues with AJAX * any code failures that result in exceptions * the obvious (catching failures/regressions in our QUnit tests)
Even code that doesn't have unit tests. The code execution alone should result an uncaught exception, which we can now get our hands on since we actually execute the javascript in a real browser. This includes anything related to ResourceLoader, since we don't just execute the unit tests in a browser, we load them from MediaWiki's core/index.php entry point (Special:JavaScriptTest, to be specific).
Similar to how we have the php-checkstyle job currently, the QUnit job is in non-voting mode. However, unlike php-checkstyle, our QUnit tests are actually passing, but we're not letting it vote yet to see how it behaves over the next 24 hours. If it is stable, we'll make it voting (like phplint, jshint and phpunit are already).
So, next time you read the jenkins-job comment, look for the QUnit job.
Happy testing,
-- Krinkle
[1] https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing
[2] PhantomJS: http://phantomjs.org/
[3] node-phantomjs (npm wrapper with nom-install hook) https://github.com/Obvious/phantomjs
[4] grunt-contrib-qunit https://github.com/gruntjs/grunt-contrib-qunit
[5] https://www.mediawiki.org/wiki/Continuous_integration/Workflow_specification
On Mar 4, 2013 7:13 PM, "Krinkle" krinklemail@gmail.com wrote:
Hey all,
As of today, we automatically run our QUnit test suite[4] in MediaWiki core from Jenkins.
Awesome, thanks Krinkle!
Greg
On Mon, Mar 4, 2013 at 8:07 PM, Ori Livneh ori@wikimedia.org wrote:
Today I sprinted to pick up QUnit testing in Jenkins and get it stabilised and deployed.
This is fantastic. Thanks, Timo.
Indeed - this is a great milestone. Thanks for all your work getting this out the door, Timo! :-)
Erik
On Tue, Mar 5, 2013 at 4:12 AM, Krinkle krinklemail@gmail.com wrote:
As of today, we automatically run our QUnit test suite[4] in MediaWiki core from Jenkins.
Great news!
I won't go in detail about what PhantomJS is, but in short:
It is a "headless" WebKit browser. Meaning, it doesn't render pixels to a screen on the server side, but it does behave like a fully valid browser environment as if it were rendering it to a screen (CSS is being parsed, the DOM is there, stylesheets are active, retrieving computed styles, ajax requests can be made etc.). For more information, see [2].
PhantomJS can render pixels, but I am not sure if it does it by default, or only when requested. The reason I know it can render pixels is that you can take a screen shot while driving PhantomJS with Selenium. Screenshots from a headless browser. Think about it for a minute. :)
Željko
On Tue, Mar 5, 2013 at 4:12 AM, Krinkle krinklemail@gmail.com wrote:
However, unlike php-checkstyle, our QUnit tests are actually passing
From console[1]:
02:50:21 Testing http://localhost:9412/mediawiki-core-28a705a9f648da310ff2a4fca9d013bf147f3d1... by test.module1: expected 02:50:21 Error: expected 02:50:24 ....................................................OK 02:50:24 >> 832 assertions passed (5350ms) 02:50:24 02:50:24 Done, without errors.
This is strange. The console says there was a problem: "Exception thrown by test.module1: expected" and "Error: expected" But then it says everything is fine: "Done, without errors."
Željko -- [1] https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console
From console[1]:
02:50:21 Testing
http://localhost:9412/mediawiki-core-28a705a9f648da310ff2a4fca9d013bf147f3d1...
by test.module1: expected 02:50:21 Error: expected 02:50:24 ....................................................OK 02:50:24 >> 832 assertions passed (5350ms) 02:50:24 02:50:24 Done, without errors.
This is strange. The console says there was a problem: "Exception thrown
by
test.module1: expected" and "Error: expected" But then it says everything is fine: "Done, without errors."
Željko
[1]
https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console
I'm not sure how to navigate to the source that defines that test but I suspect it's just using an expected exception test: http://docs.jquery.com/QUnit/raises
On Mar 5, 2013, at 3:49 PM, Dan Andreescu dandreescu@wikimedia.org wrote:
From console[1]:
02:50:21 Testing
http://localhost:9412/mediawiki-core-28a705a9f648da310ff2a4fca9d013bf147f3d1...
by test.module1: expected 02:50:21 Error: expected 02:50:24 ....................................................OK 02:50:24 >> 832 assertions passed (5350ms) 02:50:24 02:50:24 Done, without errors.
This is strange. The console says there was a problem: "Exception thrown
by
test.module1: expected" and "Error: expected" But then it says everything is fine: "Done, without errors."
Željko
[1]
https://integration.mediawiki.org/ci/job/mediawiki-core-qunit/3/console
I'm not sure how to navigate to the source that defines that test but I suspect it's just using an expected exception test: http://docs.jquery.com/QUnit/raises
No, it is neither.
Remember you're looking at the "console" log. Which, in this case is being written two from lots of sources:
* jenkins - stdout * grunt/qunit - stdout * phantomjs - console.log()
The part cited here is mostly qunit's output (the "dotted" progress line), but when calling "console.log" from within the javascript it is forwarded to the jenkins console.
A console log is harmless and no reason for alarm. If it were an actual exception, it wouldn't be tolerated.
In this case it is coming from the unit tests that asserts that mw.loader sets a module's state to "error" if it's javascript bundle throws an exception. mw.loader executes the bundle in a try/catch and logs any exceptions to the console after which it sets state=error and triggers the error callbacks.
Right now grunt/qunit forwards the phantomjs console straight to the main output. It would be prettier if it would display it in a way that more clearly says "phantomjs console". I requested this 3 months ago: https://github.com/gruntjs/grunt-contrib-qunit/pull/6
Execute the tests in your own browser and you'll see the same data in your browser's console (e.g. Chrome Dev Tools).
-- Krinkle
On 03/04/2013 07:12 PM, Krinkle wrote:
Things this will catch are basically everything else. Any runtime error that we can't detect in static analysis but will fail no matter what browser you're in, such as:
- misspelled identifiers or syntax errors
- issues with ResourceLoader (mw.loader)
- issues with AJAX
- any code failures that result in exceptions
- the obvious (catching failures/regressions in our QUnit tests)
This is really a great step forward.
Thanks,
Matt Flaschen
wikitech-l@lists.wikimedia.org