tl;dr below, frustration ahead.
I've been trying to get Jenkins to run the parsertests of LabeledSectionTransclusion, but this has been an incredibly frustrating experience. It shouldn't be too hard, right? Well, read on...
I thought I'd start with the existing 'testextensions' jenkins job [1], which is failing. The reason it is failing is because of this:
*12:02:56* + php tests/phpunit/phpunit.php --log-junit junit-phpunit-ext.xml -- extensions/LabeledSectionTransclusion*12:02:56* Unexpected non-MediaWiki exception encountered, of type "PHPUnit_Framework_Exception"*12:02:56* exception 'PHPUnit_Framework_Exception' with message 'Neither "extensions/LabeledSectionTransclusion.php" nor "extensions/LabeledSectionTransclusion.php" could be opened.' in /usr/share/php/PHPUnit/Util/Skeleton/Test.php:100
After installing PHPUnit (on itself quite a frustrating experience), I get this response instead:
valhallasw@lisilwen:~/src/core$ php tests/phpunit/phpunit.php extensions/LabeledSectionTransclusion PHPUnit 3.7.10 by Sebastian Bergmann.
Configuration read from /home/valhallasw/src/core/tests/phpunit/suite.xml
Time: 0 seconds, Memory: 18.50Mb
No tests executed!
Which suggests it might be a problem with the Jenkins configuration. Of course, the tests are also not running, but this is a second issue.
So, where is the Jenkins configuration? Not in Jenkins itself. After logging in in Jenkins, there /is/ suddenly a new option 'Workspace' (why is that disabled for logged-out users?), which shows the extension is checked out in the workspace - so that doesn't explain why phpunit is unable to locate the tests.
So let's check the configuration. [2] suggests it should be at [3]. Ugh, gitweb, and no github mirror. Under 'jobs', there are some extensions, but not LST. There is MediaWiki-Tests-Extensions, which has some XML files that, at least for me, are incomprehensible. And are they even relevant to this? In any case, they don't tell me why PHPUnit is unable to find the LST directory, even though it's right there [4]. I give up.
By the way - I just realized why the job is not in the Jenkins repository:
valhallasw@lisilwen:~/src/jenkins$ grep -R * -e 'mwext'
jobs/.gitignore:/mwext-*
*Really?*
Okay, so then how do I get PHPUnit to run the parsertests? There is the NewParserTest class, so I was guessing adding this to LabeledSectionTransclusion.php would work:
class LSTParserTest extends NewParserTest {
function __construct()
{
parent::__construct();
$this->setParserTestFile(__DIR__ . "/lstParserTests.txt");
}
};
Well, that does, er, absolutely nothing. [5] suggests I should move it to a seperate file and add the following:
$wgHooks['UnitTestsList'][] = 'efFruitsRegisterUnitTests';function efFruitsRegisterUnitTests( &$files ) { $testDir = dirname( __FILE__ ) . '/'; $files[] = $testDir . 'AppleTest.php'; return true;}
what it *doesn't* tell you is that this is actually irrelevant when you run php tests/phpunit/phpunit.php extensions/LabeledSectionTransclusion - and it's not really clear to me for when it is relevant. After some struggling, apparently it detects tests because the filename *ends in Test.php* (which is documented, well, no-where. After moving the test case there, it still didn't quite work, but after some more fiddling, something is working!
Okay, so now I have a test case that actually runs when I call the above command. So, let's push it for review [6] - but that doesn't actually make Jenkins run the test cases.... so I still don't know whether this actually fixes it.
[1] https://integration.mediawiki.org/ci/job/mwext-LabeledSectionTransclusion-te... [2] http://www.mediawiki.org/wiki/Continuous_integration/Jenkins [3] https://gerrit.wikimedia.org/r/gitweb?p=integration/jenkins.git;a=tree [4] https://integration.mediawiki.org/ci/job/mwext-LabeledSectionTransclusion-te... [5] http://www.mediawiki.org/wiki/Manual:PHP_unit_testing/Writing_unit_tests_for...
So, tl,dr: debugging jenkins failures and getting phpunit to work is a frustrating experience, because of a lack of documentation and a lack of 'obviousness' (there is not one 'obvious' way to do it). I've updated [5] with my experiences, but I still have no idea how the magic that is Jenkins is configured and how it works. It would be great if there would be better documentation on that.
Some specific things that I think should be fixed: - Jenkins should not show new (read-only) options to users - make them public instead - The Jenkins configuration should also be mirrored on github - The mwext jobs should be in git (preferrably, because then it's easy to check them on-line), *or* there should be clear documentation on how to generate those jobs... - There should be documentation on how the configuration is actually used - and preferrably on how to debug failures.
Thanks.
Merlijn
If you only want to run parser tests from a different file, there's no need to create a new class.
You simply add in the main file: $wgParserTestFiles[] = dirname( __FILE__ ) . "/lstParserTests.txt"; (already in lst.php)
It will be automagically picked when running the phpunit tests (if you have the extension enabled in LocalSettings, which would be a precondition for them to work).
You can also run them with tests/parserTests.php
With phpunit: $ make destructive
Tests: 5125, Assertions: 927178, Failures: 2, Incomplete: 3, Skipped: 5. Tests: 5125, Assertions: 880313, Failures: 2, Incomplete: 3, Skipped: 5.
Add lst to LocalSettings.
$ make destructive Tests: 5157, Assertions: 883148, Failures: 2, Incomplete: 3, Skipped: 5. Tests: 5157, Assertions: 942272, Failures: 2, Incomplete: 3, Skipped: 5. Tests: 5157, Assertions: 913264, Failures: 2, Incomplete: 3, Skipped: 5.
You can also run make parser if you prefer to run less tests. (no, I don't know why would the number of assertions randomly change between runs with the same config...)
And fyi, they do pass.
Hi Platonides,
Thanks for your response (and I extend my thanks to NikeRabbit, who improved the extension unit test page a lot!).
On 12 December 2012 00:04, Platonides Platonides@gmail.com wrote:
If you only want to run parser tests from a different file, there's no need to create a new class.
You simply add in the main file: $wgParserTestFiles[] = dirname( __FILE__ ) . "/lstParserTests.txt";
It will be automagically picked when running the phpunit tests
Yes, they are, as long as you are running the entire test suite. However, it doesn't allow you to just run a specific extensions' parser tests by calling phpunit.php on that extension's directory, which is (as far as I could find out) what Jenkins does. make parser comes close, but it also runs all other parser tests - so it runs 5000 tests instead of the 35-ish relevant ones.
You can also run them with tests/parserTests.php
This is indeed an effective way (and what I've been doing manually), (it's also the way that has been documented at http://www.mediawiki.org/wiki/Parser_tests ).
Again, thanks for your response.
Merlijn
wikitech-l@lists.wikimedia.org