Currently there are so many failures in core unit tests that it I can't run the tests locally. The new errors would only hide in between all the other failures. Most of these are caused by the tests assuming certain configuration and some issues comes from extensions.
I've reported bugs about 10 days ago but so far they have been either 1) silent 2) blaming me for having broken configuration 3) suggesting a fix.
Here is a list: https://bugzilla.wikimedia.org/show_bug.cgi?id=40491 https://bugzilla.wikimedia.org/show_bug.cgi?id=40490 https://bugzilla.wikimedia.org/show_bug.cgi?id=40489 https://bugzilla.wikimedia.org/show_bug.cgi?id=40488 https://bugzilla.wikimedia.org/show_bug.cgi?id=40487 https://bugzilla.wikimedia.org/show_bug.cgi?id=40484 [CentralNotice] https://bugzilla.wikimedia.org/show_bug.cgi?id=40483 [SMW] https://bugzilla.wikimedia.org/show_bug.cgi?id=40432 [Maps]
I don't have time to fix all these myself.
-Niklas
FAILURES! Tests: 4685, Assertions: 320062, Failures: 62, Errors: 45, Incomplete: 5, Skipped: 6. make: *** [safe] Error 2
While we're at it, the tests for the REL1_18 branch in git are also broken. It would be great to get them fixed, so Jenkins can run on that branch.
On Thu, Oct 4, 2012 at 12:44 AM, Niklas Laxström niklas.laxstrom@gmail.com wrote:
Currently there are so many failures in core unit tests that it I can't run the tests locally. The new errors would only hide in between all the other failures. Most of these are caused by the tests assuming certain configuration and some issues comes from extensions.
I've reported bugs about 10 days ago but so far they have been either
- silent 2) blaming me for having broken configuration 3) suggesting
a fix.
Here is a list: https://bugzilla.wikimedia.org/show_bug.cgi?id=40491 https://bugzilla.wikimedia.org/show_bug.cgi?id=40490 https://bugzilla.wikimedia.org/show_bug.cgi?id=40489 https://bugzilla.wikimedia.org/show_bug.cgi?id=40488 https://bugzilla.wikimedia.org/show_bug.cgi?id=40487 https://bugzilla.wikimedia.org/show_bug.cgi?id=40484 [CentralNotice] https://bugzilla.wikimedia.org/show_bug.cgi?id=40483 [SMW] https://bugzilla.wikimedia.org/show_bug.cgi?id=40432 [Maps]
I don't have time to fix all these myself.
-Niklas
FAILURES! Tests: 4685, Assertions: 320062, Failures: 62, Errors: 45, Incomplete: 5, Skipped: 6. make: *** [safe] Error 2
-- Niklas Laxström
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Are you planning that for before, or after 1.18 is made obsolete in 27 days. ;)
It would be nice if jenkins also did a run with some non-default options. Obviously we have a lot of options, but doing at least 1 run with some of the common variations (say $wgCapitalLinks) would help quite a bit I imagine.
-bawolff
Le 05/10/12 19:11, bawolff a écrit :
It would be nice if jenkins also did a run with some non-default options. Obviously we have a lot of options, but doing at least 1 run with some of the common variations (say $wgCapitalLinks) would help quite a bit I imagine.
The Jenkins job setup a default MediaWiki install which mean it is mostly using DefaultSettings.php. If we want to test wgCapitalLinks, we should write tests for that feature and adapt existing tests to take in account that global variable.
On Fri, 05 Oct 2012 12:17:41 -0700, Antoine Musso hashar+wmf@free.fr wrote:
Le 05/10/12 19:11, bawolff a écrit :
It would be nice if jenkins also did a run with some non-default options. Obviously we have a lot of options, but doing at least 1 run with some of the common variations (say $wgCapitalLinks) would help quite a bit I imagine.
The Jenkins job setup a default MediaWiki install which mean it is mostly using DefaultSettings.php. If we want to test wgCapitalLinks, we should write tests for that feature and adapt existing tests to take in account that global variable.
The same case probably goes for a number of our other config variables.
There are a few places where we go and swap out the value of a config variable and then reset it. Last one I saw was wgHtml5 in HtmlTests. However even that is almost never actually used. We don't bother testing both situations.
We should probably introduce some code to make tweaking and then resetting config in tests easy so we can write more of our tests to check different modes of config.
Le 05/10/12 21:40, Daniel Friesen wrote:
The same case probably goes for a number of our other config variables.
There are a few places where we go and swap out the value of a config variable and then reset it. Last one I saw was wgHtml5 in HtmlTests. However even that is almost never actually used. We don't bother testing both situations.
We should probably introduce some code to make tweaking and then resetting config in tests easy so we can write more of our tests to check different modes of config.
PHPUnit comes with a feature to automatically backup globals though we always had it disabled ( tests/phpunit/suite.xml has backupglobals=false). I have never investigated the use of this mechanism, that could potentially save us some troubles.
On Oct 5, 2012, at 9:17 PM, Antoine Musso hashar+wmf@free.fr wrote:
Le 05/10/12 19:11, bawolff a écrit :
It would be nice if jenkins also did a run with some non-default options. Obviously we have a lot of options, but doing at least 1 run with some of the common variations (say $wgCapitalLinks) would help quite a bit I imagine.
The Jenkins job setup a default MediaWiki install which mean it is mostly using DefaultSettings.php. If we want to test wgCapitalLinks, we should write tests for that feature and adapt existing tests to take in account that global variable.
-- Antoine "hashar" Musso
I agree. Tests should make sure their expected results are based on the configuration they set in the setup/teardown (as opposed to relying on the default configuration). And if there are any significant differences possible in the output based on configuration options, then the test should provide tests for that (as opposed to re-running all test suites for one configuration option which is an approach full for wasting resources and doesn't scale).
So the Html test cases should just set $wgHtml5=true and $wgWellFormedXml and be done with it. Because that it will add /> everywhere is obvious (it should have one or two assertions to make sure the other behaviour is also tested), no need to duplicate the entire test suite 2 or 3 times for unimportant details.
-- Krinkle
wikitech-l@lists.wikimedia.org