[QA] Helper to run MediaWiki tests

[Antoine Musso]

On 13/04/2018 10:48, Kunal Mehta wrote:
> I LOVE IT
> 
> But really, I was failing to reproduce a test failure[1] locally and I
> gave quibble a try and it failed! After a bit of playing around I was
> able to do a full git bisect, pinpointing the problem.
> 
> My main feature request would be to be able to run each "stage"
> independently, e.g. --stage=qunit or --stage=phpunit or
> --stage=composer-test, etc. I'd think we'd need that anyways to keep our
> current parallelization of jobs.
> 
> [1] https://phabricator.wikimedia.org/T191863

Thank you Kunal for your reply last week, that has encouraged me all
week long (as well as Timo praised).

Quibble 0.0.8 has your patch to select which test command to run using
--run. That is definitely an improvement when testing locally, I had
just commented out blocks here and there in cmd.py :-/


We run tests against several environments varying based on the php
flavor, database or vendor vs composer.   I thought about having a
reference environment for a given branch, and when varying when of the
input (eg: database = sqlite), only run commands it affects.

I did a very basic pass https://gerrit.wikimedia.org/r/#/c/427099/ which
more or less let us handle the variations and skip unwanted commands. It
is not any ready though and I am not sure where I am going on that front.

Side note: I have added quibble to mediawiki/core yesterday/today. Going
to announce it to wikitech-l right now.


-- 
Antoine "hashar" Musso