[QA] Helper to run MediaWiki tests

[Kunal Mehta]

I LOVE IT

But really, I was failing to reproduce a test failure[1] locally and I
gave quibble a try and it failed! After a bit of playing around I was
able to do a full git bisect, pinpointing the problem.

My main feature request would be to be able to run each "stage"
independently, e.g. --stage=qunit or --stage=phpunit or
--stage=composer-test, etc. I'd think we'd need that anyways to keep our
current parallelization of jobs.

[1] https://phabricator.wikimedia.org/T191863

-- Legoktm

On 04/05/2018 01:13 PM, Antoine Musso wrote:
> Hello,
> 
> I would like to introduce Quibble, a small script intended to easily run
> all MediaWiki tests with just a few keystrokes.  It will ultimately be
> used to power up all the CI jobs having to deal with MediaWiki.
> 
> During the Vienna hackathon (2017), we had a lot of discussions
> regarding Docker and how to reproduce what is being run by CI.  I wrote
> a quick proof of concept that was able to run the MediaWiki PHPUnit
> tests as well as the QUnit ones. For the last three weeks I came back to
> the POC and largely polished it to a point in confident in switching to it.
> 
> Given an environment providing a PHP flavor, NodeJS, chromium,
> composer/npm and a database (MySQL, MariaDB, sqlite), Quibble setups
> MediaWiki and then runs all tests commands you can imagine of.
> 
> To install Quibble:
> 
>  git clone https://gerrit.wikimedia.org/r/p/integration/quibble
>  cd quibble
>  python3 -m pip install -e .
> 
> Then eventually one can just:
> 
>  quibble
> 
> Or try some other alternatives:
> 
>  ZUUL_BRANCH=REL1_30 quibble --db sqlite --packages-source composer
> 
> 
> Refer to the README.md for some more basic setup, specially to setup a
> cache of git repositories.  There is a Dockerfile available at the root
> of the repository for some quick testing, and a reference Docker
> container intended for CI in integration/config:
> 
> docker-registry.discovery.wmnet/releng/quibble-stretch:latest
> 
> 
> The default workflow is:
> 
> * clone mediawiki/core and mediawiki/skins/Vector
> * clone mediawiki/vendor
> * spawn mysqld (only tested with MariaDB)
> * run install.php
> * composer require mediawiki/core devDependencies
> * npm install
> 
> At this point the basics are set. Quibble then runs:
> 
> * phpunit excluding tests in the 'Database' group
> * composer test, passing it solely files changed in HEAD
> * npm test
> 
> * spawn a webserver using maintenance/dev/includes/router.php
> * qunit  (grunt karma:main)
> * if no DISPLAY is available, spawn Xvfb and Chromedriver pointing to it
> * the Webdriver.io suite (grunt webdriver:test)
> 
> * Finally the PHPUnit Database tests.
> 
> A full run in Jenkins takes roughly 6 minutes with a warm cache.
> 
> 
> There is some support to install extensions and run their tests
> (phpunit.php --testsuite extensions).
> 
> I have added an experimental job to mediawiki/core mediawiki/vendor and
> mediawiki/skins/Vector. I will make sure it passes on master and start
> switching to it.
> 
> The job is mediawiki-core-quibble-vendor-mysql-php7-docker in the
> 'experimental' pipeline:
> https://integration.wikimedia.org/ci/job/mediawiki-core-quibble-vendor-mysql-php7-docker/
> 
> Please give it a try and send issues, support requests to Phabricator:
> https://phabricator.wikimedia.org/tag/quibble
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.wikimedia.org/pipermail/qa/attachments/20180413/fdd7b882/attachment.sig>