TL;DR:
* Our QUnit jobs now uses latest Chromium instead of PhantomJS.
* You can run the test suite from the command line locally now.
Thanks to Tim Starling, Antoine "hashar" Musso, Kunal (legoktm),
Bryan Davis, S Page, and James Forrester; for their help in different areas.
https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing
Are you sitting comfortably?
The past few months can be summarised as a long journey through a forest of technical
debt. It was also amazing to see just how many layers of infrastructure were able to block
this task (and did). [1]
From a local development point of view it couldn't be simpler. In the Gruntfile:
Remove grunt-contrib-qunit, add grunt-karma with karma-chrome-launcher. Done? Not quite.
For standalone front-end projects it actually was this simple. OOjs and VisualEditor have
been enjoying this new stack since July 2014. This helped refine the stack and the
underlying technologies over the past months. More about what this new stack provides in a
minute.
== Journey ==
=== Export from Special:JavaScriptTest ===
For MediaWiki core and extensions, one must install MediaWiki before running QUnit. [2][3]
That in itself isn't too complicated (set up DB and run install; we have standard
macros for that). But, the test doesn't just communicate with MediaWiki. The test
suite is actually served *by* MediaWiki. Karma enforces the principle that unit tests run
in pure javascript (blank page with sockets, load source files, load QUnit, run test
suite).
Back in MediaWiki 1.19 our test suite abided by these best practices. The test suite was a
static HTML file that loaded relevant scripts directly. [4] This file was later migrated
to Special:JavaScriptTest, which opened the door for undeclared dependencies.
1. Source code is registered to ResourceLoader.
2. Extensions register tests via hooks in PHP.
3. OutputPage and Skin provide config vars and HTML that tests could depend on.
Point 3 was easily resolved by adding the appropriate mw.config or DOM fixture to the few
tests that were missing it.
Though Point 2 and 1 weren't going anywhere. We needed to, once again, access our
tests suite as a pure JavaScript payload, with no HTML or script tags loading relevant
resources.
Hence, the introduction of Special:JavaScriptTest/qunit/export in
https://gerrit.wikimedia.org/r/178551.
=== Migrate all the things ===
Objective: Karma [5] on Chromium. Easy to run locally for developers.
To be re-usable, the logic must be generic and use composer/npm. In 2013, most jobs ran on
Wikimedia production servers. In order to use npm we had to migrate to Wikimedia Labs
first.
Back then, production had only just begun using Ubuntu Trusty.
The relevant modules require Node.js v0.10. Ubuntu Precise has Node.js v0.8. Precise is
also stuck with Chromium 37 (EOL). [7] To get auto-updating latest stable Chromium and
Node v0.10, we needed to migrate our infrastructure to Ubuntu Trusty first.
Much of the process to install MediaWiki on a Jenkins slave also wasn't puppetised.
=== SQLite ===
This is its own story. See
https://phabricator.wikimedia.org/T89180 for details.
The short of it is, we also migrated everything to MySQL. Which we wanted to do anyway.
== New stack ==
1. Chromium
PhantomJS is a great application for many purposes. It was good for us while it lasted.
But for JavaScript unit testing, PhantomJS just isn't meant to be. It's too
distant from a "real" browser. When we introduced PhantomJS, it was a big step
forward toward testing cross-browser. It "uses WebKit" which meant we were kind
of covering Chrome and Safari (in 2012).
Safari has had several major releases since PhantomJS v1.9 came out. Chrome had even more
releases (and since dropped WebKit in favour of Blink). And, in truth, PhantomJS
wasn't that much like Safari and mainstream WebKit to begin with. [8]
Chromium is an actual browser. A browser we actually support. A browser that represents
real users of our software. It's only one browser for now, but it's a start.
2. Karma
With Karma as a solid foundation, it's truly simple to add more browsers in the
future. OOjs has already added Firefox alongside Chromium in Jenkins. Karma uses Web
Driver, a standardised protocol to operate browsers (locally and remotely alike). It has
an official plugin hooking it up to SauceLabs, which opens the door to any other browser
and platform we want. It's as simple as adding 1 line to the config. [5][6]
We haven't done this yet as there are scalability, performance and security
considerations. Having said that, OOjs is currently using an experimental pipeline to test
in six different browsers from Jenkins via SauceLabs. Including IE 6 and IE 11 on Windows,
and Safari 5 on Mac! [9]
3. Code coverage
One of the advantages of having a pure stack is the ability to run pre-processors on the
code using Karma built-ins. [10] The first one I'm looking at is code coverage.
4. Local development and Jenkins equally
It runs the full QUnit test suite in a real browser (multiple even) by simply running
"npm test". [11]
Aside from Node.js, it requires no pre-installed software and is as easy as "npm
install" to set up.
— Krinkle
[1]
https://phabricator.wikimedia.org/T74063 [epic] Adopt Karma with Chromium (tracking).
[2]
https://phabricator.wikimedia.org/T89433 Make QUnit tests run without installing
MediaWiki.
[3]
https://phabricator.wikimedia.org/T89432 Make PHPUnit tests run without installing
MediaWiki.
[4]
https://github.com/wikimedia/mediawiki/blob/REL1_19/tests/qunit/index.html
[5]
https://karma-runner.github.io/
[6]
https://karma-runner.github.io/0.12/config/browsers.html
[7]
http://packages.ubuntu.com/precise/chromium-browser
[8]
http://codepen.io/Krinkle/blog/phantomjs-anno-2014
[9]
https://github.com/wikimedia/oojs/blob/v1.1.6/Gruntfile.js#L76-L107
https://integration.wikimedia.org/ci/job/npm/1680/console
[11]
https://karma-runner.github.io/0.12/config/preprocessors.html
https://github.com/karma-runner/karma-coverage
[12]
https://www.mediawiki.org/wiki/Manual:JavaScript_unit_testing