Dear Wikimedia Developers,
tl;dr: as part of continuing efforts [1][2] to speed up our CI jobs, we have experimentally enabled parallel PHPUnit tests for WMDE maintained extensions[3]. In experimental testing, this reduces the time taken to run the longest CI tests from 25 minutes to around 8 minutes. If you see any problems, please report them on Phabricator at https://phabricator.wikimedia.org/T361190.
Arthur
Longer version
--------------------
The parallel testing implementation uses PHPUnit’s `--list-tests` function to generate a list of test classes, then creates 7-8 “buckets” of test classes which are executed as suites in parallel. Experimentally, this reduces the time taken to run the longest CI tests from 25 minutes to around 8 minutes.
We are rolling this functionality out progressively as we have some concerns about its impact. We want to make the developer community aware of our concerns and ask you to help us spot potential issues as they occur. There are three main concerns:
1. Overloading the CI infrastructure
-----------------------------------------------
Having the job runners execute the tests in parallel means that fewer runners will be using more CPUs. This may increase queuing time during busy periods, or may lead to partial or complete denial of service depending on how the jobs stack up and how the infrastructure copes with the new load profile.
2. Unexpected test failures
-----------------------------------
Splitting the test suite and running the classes in a random order can surface bad interactions between test cases that we didn’t see before - we have a lot of global state in the system, and not all test classes are careful in their use of `setUp` and `tearDown` methods. If you suddenly start to see failures that were not present before, or you see failures that you are unable to reproduce on your local machine, that might be a result of the tests being run in a new combination because of the parallel execution. If you see this, please file a task and make it a subtask of T361190.
3. Incomplete test runs
------------------------------
Because of how we are extracting the list of tests from PHPUnit and constructing a new suite, it may be the case that we “lose” test classes from the Parallel run that were present in the serialized execution. We have done some testing to increase our confidence that this is not the case, but it is still a potential risk. If you think some tests are not being run, please file a task and make it a subtask of T361190.
How do I report a problem?
------------------------------------
The changes were deployed today (June 25th) at 14h24 CEST. If you start to notice problems with your CI jobs that were not there before, please let us know with a comment on https://phabricator.wikimedia.org/T361190 . If the problems are urgent and we need to roll back the change quickly, you can contact Arthur (codders), Kosta (kostajh) or Antoine (hashar) to get the parallel change reverted, in #wikimedia-releng.
What’s next
---------------
The current implementation is Python-based and all the changes are in Quibble/Zuul - these changes only affect CI jobs. To help developers speed up their local testing, we are also working on a PHP-/composer-based implementation [4], and when that is mature we will switch from the Python- to the PHP-based implementation in CI.
We have also looked at and continue to be interested in an implementation based on paratest[5]. Our current test suite is complex in ways that paratest does not currently support, and it looks like the support we need will only be available when we have migrated our test suite to PHPUnit 10.
References:
1.
https://phabricator.wikimedia.org/T361190 2.
https://phabricator.wikimedia.org/T50217 3.
https://gerrit.wikimedia.org/r/c/integration/config/+/1047919, 1.
AdvancedSearch 2.
ArticlePlaceholder 3.
Cognate 4.
ElectronPdfService 5.
EntitySchema 6.
FileExporter 7.
FileImporter 8.
InterwikiSorting 9.
PropertySuggester 10.
RevisionSlider 11.
TwoColConflict 12.
Wikibase 13.
Wikidata.org 14.
WikibaseLexeme 15.
WikimediaBadges 4.
https://phabricator.wikimedia.org/T365567 5.
https://github.com/paratestphp/paratest