[QA] Helper to run MediaWiki tests

[Antoine Musso]

Hello,

I would like to introduce Quibble, a small script intended to easily run
all MediaWiki tests with just a few keystrokes.  It will ultimately be
used to power up all the CI jobs having to deal with MediaWiki.

During the Vienna hackathon (2017), we had a lot of discussions
regarding Docker and how to reproduce what is being run by CI.  I wrote
a quick proof of concept that was able to run the MediaWiki PHPUnit
tests as well as the QUnit ones. For the last three weeks I came back to
the POC and largely polished it to a point in confident in switching to it.

Given an environment providing a PHP flavor, NodeJS, chromium,
composer/npm and a database (MySQL, MariaDB, sqlite), Quibble setups
MediaWiki and then runs all tests commands you can imagine of.

To install Quibble:

 git clone https://gerrit.wikimedia.org/r/p/integration/quibble
 cd quibble
 python3 -m pip install -e .

Then eventually one can just:

 quibble

Or try some other alternatives:

 ZUUL_BRANCH=REL1_30 quibble --db sqlite --packages-source composer


Refer to the README.md for some more basic setup, specially to setup a
cache of git repositories.  There is a Dockerfile available at the root
of the repository for some quick testing, and a reference Docker
container intended for CI in integration/config:

docker-registry.discovery.wmnet/releng/quibble-stretch:latest


The default workflow is:

* clone mediawiki/core and mediawiki/skins/Vector
* clone mediawiki/vendor
* spawn mysqld (only tested with MariaDB)
* run install.php
* composer require mediawiki/core devDependencies
* npm install

At this point the basics are set. Quibble then runs:

* phpunit excluding tests in the 'Database' group
* composer test, passing it solely files changed in HEAD
* npm test

* spawn a webserver using maintenance/dev/includes/router.php
* qunit  (grunt karma:main)
* if no DISPLAY is available, spawn Xvfb and Chromedriver pointing to it
* the Webdriver.io suite (grunt webdriver:test)

* Finally the PHPUnit Database tests.

A full run in Jenkins takes roughly 6 minutes with a warm cache.


There is some support to install extensions and run their tests
(phpunit.php --testsuite extensions).

I have added an experimental job to mediawiki/core mediawiki/vendor and
mediawiki/skins/Vector. I will make sure it passes on master and start
switching to it.

The job is mediawiki-core-quibble-vendor-mysql-php7-docker in the
'experimental' pipeline:
https://integration.wikimedia.org/ci/job/mediawiki-core-quibble-vendor-mysql-php7-docker/

Please give it a try and send issues, support requests to Phabricator:
https://phabricator.wikimedia.org/tag/quibble

-- 
Antoine "hashar" Musso