Hey all,
The idea of separating PHPUnit unit and integration/system tests in MediaWiki core has been around for some time[1][2][3]. Currently, the tests assume the presence of valid MediaWiki settings and a database connection, meaning one must install & configure MediaWiki and an RDBMS in their local development environment to be able to run the tests. The fact we use a non-standard entry point (phpunit.php) also makes these tests incompatible with existing tooling such as IDE integrations.[4]
At the 2019 hackathon I worked with Amir Sarabadani, Michael Große and Kosta Harlan to perform some preliminary investigation into separating unit tests (that can be run without a database and MediaWiki configuration) into a separate PHPUnit configuration that could be run via the official phpunit binary. After some additional work, this has evolved into a patch that separates 5301 unit tests into a dedicated suite that can be executed via vendor/bin/phpunit in 15 seconds (on my machine!). By contrast, running the same 5301 tests via phpunit.php takes around 30 seconds.[5] Not using phpunit.php here also allows for integrating with e.g. Intellij/PHPStorm’s test execution and code coverage functionality—here’s a screenshot of the execution and coverage information of PathRouterTest via IntelliJ.[6] I feel that these benefits would be felt both by developers and CI maintainers—developers could iterate more rapidly by running the unit tests, and Jenkins would have to spend less time executing the test suite.
I’d like to thank everyone who has supported this enterprise so far—Amir (Ladsgroup) for creating a script to identify the initial set of tests that do not rely on a database[7][8] and providing assistance throughout, Kosta Harlan for highlighting this old and forgotten issue and bringing it to the hackathon, Bartosz Dziewoński (MatmaRex) for providing a solution to scoping issues around MediaWiki core files,[9] Michael Große for demonstrating the feasibility of this approach in the Wikibase extension test suite,[10] and James Forrester for reviewing the changes and outlining possible next steps.
However, the work is far from done yet! The patch is not yet merged, so any reviews, comments and suggestions would be very welcome there! And if it does get merged, we will have to think about how to bring this separation to extensions’ test suites as well as potentially port more core tests to the unit test suite. So if you have any ideas on how to improve this patch or would like to add to the next steps, don’t hesitate to leave a note :)
[1] https://phabricator.wikimedia.org/T89432 [2] https://phabricator.wikimedia.org/T87781 [3] https://phabricator.wikimedia.org/T84948 [4] https://www.mediawiki.org/wiki/Manual:PHP_unit_testing/Running_the_unit_test... [5] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/513106 [6] https://phabricator.wikimedia.org/F29316124 [7] https://phabricator.wikimedia.org/T87781#5194643 [8] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/510226/ [9] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/513106#message-32b2a171a7c... [10] https://gerrit.wikimedia.org/r/511035
Yours, --- Máté Szabó Software Engineer +36 30 947 5903
WIKIA sp. z o.o. z siedzibą w Poznaniu, ul. Abp. A. Baraniaka 6 Sąd Rejonowy Poznań – Nowe Miasto i Wilda w Poznaniu, VIII Wydział Gospodarczy Krajowego Rejestru Sądowego, KRS 0000254365 NIP: 5252358778 Kapitał zakładowy: 50.000,00 złotych
Thank you for all your work so far on this!
On Sun, 2 Jun 2019 at 14:58, Máté Szabó mszabo@wikia-inc.com wrote:
Hey all,
The idea of separating PHPUnit unit and integration/system tests in MediaWiki core has been around for some time[1][2][3]. Currently, the tests assume the presence of valid MediaWiki settings and a database connection, meaning one must install & configure MediaWiki and an RDBMS in their local development environment to be able to run the tests. The fact we use a non-standard entry point (phpunit.php) also makes these tests incompatible with existing tooling such as IDE integrations.[4]
At the 2019 hackathon I worked with Amir Sarabadani, Michael Große and Kosta Harlan to perform some preliminary investigation into separating unit tests (that can be run without a database and MediaWiki configuration) into a separate PHPUnit configuration that could be run via the official phpunit binary. After some additional work, this has evolved into a patch that separates 5301 unit tests into a dedicated suite that can be executed via vendor/bin/phpunit in 15 seconds (on my machine!). By contrast, running the same 5301 tests via phpunit.php takes around 30 seconds.[5] Not using phpunit.php here also allows for integrating with e.g. Intellij/PHPStorm’s test execution and code coverage functionality—here’s a screenshot of the execution and coverage information of PathRouterTest via IntelliJ.[6] I feel that these benefits would be felt both by developers and CI maintainers—developers could iterate more rapidly by running the unit tests, and Jenkins would have to spend less time executing the test suite.
I’d like to thank everyone who has supported this enterprise so far—Amir (Ladsgroup) for creating a script to identify the initial set of tests that do not rely on a database[7][8] and providing assistance throughout, Kosta Harlan for highlighting this old and forgotten issue and bringing it to the hackathon, Bartosz Dziewoński (MatmaRex) for providing a solution to scoping issues around MediaWiki core files,[9] Michael Große for demonstrating the feasibility of this approach in the Wikibase extension test suite,[10] and James Forrester for reviewing the changes and outlining possible next steps.
However, the work is far from done yet! The patch is not yet merged, so any reviews, comments and suggestions would be very welcome there! And if it does get merged, we will have to think about how to bring this separation to extensions’ test suites as well as potentially port more core tests to the unit test suite. So if you have any ideas on how to improve this patch or would like to add to the next steps, don’t hesitate to leave a note :)
[1] https://phabricator.wikimedia.org/T89432 [2] https://phabricator.wikimedia.org/T87781 [3] https://phabricator.wikimedia.org/T84948 [4] https://www.mediawiki.org/wiki/Manual:PHP_unit_testing/Running_the_unit_test... [5] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/513106 [6] https://phabricator.wikimedia.org/F29316124 [7] https://phabricator.wikimedia.org/T87781#5194643 [8] https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/510226/ [9] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/513106#message-32b2a171a7c... [10] https://gerrit.wikimedia.org/r/511035
Yours,
Máté Szabó Software Engineer +36 30 947 5903
WIKIA sp. z o.o. z siedzibą w Poznaniu, ul. Abp. A. Baraniaka 6 Sąd Rejonowy Poznań – Nowe Miasto i Wilda w Poznaniu, VIII Wydział Gospodarczy Krajowego Rejestru Sądowego, KRS 0000254365 NIP: 5252358778 Kapitał zakładowy: 50.000,00 złotych
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org