Now that tests need +2 to be run, at least temporarily, I'm going to point out that I've not been able to run tests on my development environment in ages. I mentioned broken unit tests in Oct 4 on this list. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/64390
There are multiple fatal bugs (not to mention the numerous test failures), that halt test runs without any info expect the error. Some bugs I've reported:
* https://bugzilla.wikimedia.org/41491 * https://bugzilla.wikimedia.org/42145 (skip the first few comments)
Today I tried again and there is new one:
Catchable fatal error: Argument 2 passed to OutputPage::addWikiTextTitle() must be an instance of Title, null given, called in /www/dev.translatewiki.net/w/includes/OutputPage.php on line 1426 and defined in /www/dev.translatewiki.net/w/includes/OutputPage.php on line 1472
This might be just a variant of 42145, but I can't tell for sure. I could add exception there, but the other fatal errors make phpunit not to display backtraces. I haven't yet had time to try to find out which test it is.
This situation is starting to feel like a bad horror movie, so I ask everyone to give some tender, love and care to our unit tests so that I don't have to come up with even worse analogies.
-Niklas
-- Niklas Laxström
Le 07/12/12 19:13, Niklas Laxström a écrit :
Now that tests need +2 to be run, at least temporarily, I'm going to point out that I've not been able to run tests on my development environment in ages. I mentioned broken unit tests in Oct 4 on this list. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/64390
There are multiple fatal bugs (not to mention the numerous test failures), that halt test runs without any info expect the error.
<snip>
Hello Niklas,
I will be more than happy to help you track down the root cause of such failures. It seems your setup has lot of extensions loaded in and most probably a non default LocalSettings.php.
To track the issue down you could start by disabling all extensions and see how it goes, then enable extension one by one and rerun tests in between.
Same goes for settings, try out with a fresh install of just MediaWiki core then add the settings one by one and see what happens.
cheers,
On Fri, Dec 7, 2012 at 1:13 PM, Niklas Laxström niklas.laxstrom@gmail.com wrote:
There are multiple fatal bugs (not to mention the numerous test failures), that halt test runs without any info expect the error. Some bugs I've reported:
While that unit test mentioned there does seem screwed up, why is your PHPUnit installation not respecting convertErrorsToExceptions="true" and related settings in MediaWiki's provided suite.xml?
On 7 December 2012 22:51, Brad Jorsch bjorsch@wikimedia.org wrote:
While that unit test mentioned there does seem screwed up, why is your PHPUnit installation not respecting convertErrorsToExceptions="true" and related settings in MediaWiki's provided suite.xml?
I honestly don't know. I barely got working PHPUnit installed in the first place (3.7.8 - see my other previous phpunit thread). I also don't know why it occasionally segfaults.
-Niklas
-- Niklas Laxström
They mostly run for me.
Those failing are: - PreferencesTest and TitleMethodsTest requiring a DB but not in Database group. - SanitizerTest::testRemovehtmltagsOnHtml5Tags skipping the closing tag an <data>, <mark> and <time> (???) - IPTCTest::testIPTCParseForcedUTFButInvalid with php 5.4. Reported in June as bug 37665
Plus an odd error by DatabaseTest::testStoredFunctions
On 8 December 2012 21:22, Platonides Platonides@gmail.com wrote:
They mostly run for me.
Yep, mostly: Tests: 4653, Assertions: 556494, Failures: 119, Errors: 17, Incomplete: 3, Skipped: 15.
I've not even looked at most of these failures because of the fatal errors. Thanks to Jeroen and Brad there is now one fatal error less (and at least one remains), though I did spend two hours debugging that issue. -Niklas
-- Niklas Laxström
On 12/07/2012 01:13 PM, Niklas Laxström wrote:
Now that tests need +2 to be run, at least temporarily, I'm going to point out that I've not been able to run tests on my development environment in ages. I mentioned broken unit tests in Oct 4 on this list. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/64390
Niklas, are you still having these problems, or are they mostly resolved?
The ideal is for all the regular developers to automatically run the test suite locally before submitting changesets, so if anyone's had problems that stop them from doing that, we ought to learn why, and fix those obstacles.
On Thu, Dec 27, 2012 at 11:48 PM, Sumana Harihareswara sumanah@wikimedia.org wrote:
On 12/07/2012 01:13 PM, Niklas Laxström wrote:
Now that tests need +2 to be run, at least temporarily, I'm going to point out that I've not been able to run tests on my development environment in ages. I mentioned broken unit tests in Oct 4 on this list. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/64390
Niklas, are you still having these problems, or are they mostly resolved?
The ideal is for all the regular developers to automatically run the test suite locally before submitting changesets, so if anyone's had problems that stop them from doing that, we ought to learn why, and fix those obstacles.
I used to run unit tests at regular intervals. I stopped because it is such a pain to install a version of phpunit that actually works (Also its less critical to do it yourself now that jenkins does it for you).
When I used to run unit tests, there were quite regularly issues where the unit tests assumed you had the default configuration, where they really should not assume such a thing. (That was of course a while ago, so things may have changed).
--bawolff
On Sat, Dec 29, 2012 at 8:57 AM, bawolff bawolff+wn@gmail.com wrote:
When I used to run unit tests, there were quite regularly issues where the unit tests assumed you had the default configuration, where they really should not assume such a thing. (That was of course a while ago, so things may have changed).
This is an annoyance to me as well. So, I went triaging, and finally found the issues that failed the unit tests for me. I have committed fixes for them to gerrit: https://gerrit.wikimedia.org/r/#/c/41362/ https://gerrit.wikimedia.org/r/#/c/41360/
Bryan
On Sat, Dec 29, 2012 at 1:47 PM, Bryan Tong Minh bryan.tongminh@gmail.comwrote:
This is an annoyance to me as well. So, I went triaging, and finally found the issues that failed the unit tests for me. I have committed fixes for them to gerrit: https://gerrit.wikimedia.org/r/#/c/41362/ https://gerrit.wikimedia.org/r/#/c/41360/
This sort of thing has come up before. Michael Feathers, author of Working Effectively with Legacy Code ( http://www.artima.com/weblogs/viewpost.jsp?thread=126923), etc. published a guideline for unit tests in 2005: http://www.artima.com/weblogs/viewpost.jsp?thread=126923:
A test is not a unit test if:
It talks to the database It communicates across the network It touches the file system It can't run at the same time as any of your other unit tests You have to do special things to your environment (such as editing config files) to run it.
I am not a great writer of unit tests, especially in PHP, but it is my impression that we have quite a few existing tests that don't follow this guide. In Bryan's example above, having a unit test depend on the existence of gzip would be a code smell.
I don't think it's an immediate priority, but over time it would probably make sense to refactor the tests that don't follow this guide to use mocks and stubs and other accepted practices to avoid requiring a particular environment to run unit tests. -Chris
wikitech-l@lists.wikimedia.org