An automated run of parserTests.php showed the following failures:
Running test TODO: Table security: embedded pipes (http://mail.wikipedia.org/pipermail/wikitech-l/2006-April/034637.html)... FAILED! Running test TODO: Link containing double-single-quotes '' (bug 4598)... FAILED! Running test TODO: Template with thumb image (with link in description)... FAILED! Running test TODO: message transform: <noinclude> in transcluded template (bug 4926)... FAILED! Running test TODO: message transform: <onlyinclude> in transcluded template (bug 4926)... FAILED! Running test BUG 1887, part 2: A <math> with a thumbnail- math enabled... FAILED! Running test TODO: HTML bullet list, unclosed tags (bug 5497)... FAILED! Running test TODO: HTML ordered list, unclosed tags (bug 5497)... FAILED! Running test TODO: HTML nested bullet list, open tags (bug 5497)... FAILED! Running test TODO: HTML nested ordered list, open tags (bug 5497)... FAILED! Running test Fuzz testing: image with bogus manual thumbnail... FAILED! Running test TODO: Parsing optional HTML elements (Bug 6171)... FAILED! Running test TODO: Inline HTML vs wiki block nesting... FAILED! Running test TODO: Mixing markup for italics and bold... FAILED! Running test TODO: 5 quotes, code coverage +1 line... FAILED! Running test TODO: HTML Hex character encoding.... FAILED! Running test TODO: dt/dd/dl test... FAILED!
Passed 422 of 439 tests (96.13%) FAILED!
brion@pobox.com wrote:
Passed 422 of 439 tests (96.13%) FAILED!
I've made a couple of additions to the parser test system:
First, the results can now be stored in the database for comparison between runs. This means that new tests, fixes, and regressions can be more visibly marked.
At the moment the reporting change is simply to list the number of new/removed/changed tests, but this could be improved further to give more prominence to specific regressions.
Second, the test suite can run from multiple source files. Extensions with their own test files can append them to $wgParserTestFiles, and they'll get automatically run alongside the main tests.
It might be nice to break up the big test file into smaller, more manageable pieces.
If you want to try the recording mode yourself, source the tables from maintenance/testRunner.sql and pass the --record option to parserTests.php.
Note that the recording system requires each test to have a unique name; this isn't currently enforced when not recording, but assume it will be. :)
-- brion vibber (brion @ pobox.com)
Mark Clements wrote:
brion@pobox.com wrote: Passed 422 of 439 tests (96.13%) FAILED!
That line has always puzzled me... it appears to say that 96.13% of tests failed...
The final "FAILED!" indicates that not all tests passed.
In a perfect world, you'd see 100% PASSED! :)
-- brion vibber (brion @ pobox.com)
On 11/11/06, Brion Vibber brion@pobox.com wrote:
Mark Clements wrote:
brion@pobox.com wrote: Passed 422 of 439 tests (96.13%) FAILED!
That line has always puzzled me... it appears to say that 96.13% of tests failed...
The final "FAILED!" indicates that not all tests passed.
Adding the "..." as in the lines above would indicate that "422 of 439 tests failed" is not the preferred reading.
Mathias
Mathias Schindler wrote:
On 11/11/06, Brion Vibber brion@pobox.com wrote:
Mark Clements wrote:
brion@pobox.com wrote: Passed 422 of 439 tests (96.13%) FAILED!
That line has always puzzled me... it appears to say that 96.13% of tests failed...
The final "FAILED!" indicates that not all tests passed.
Adding the "..." as in the lines above would indicate that "422 of 439 tests failed" is not the preferred reading.
Indeed; done!
-- brion vibber (brion @ pobox.com)
Second, the test suite can run from multiple source files. Extensions with their own test files can append them to $wgParserTestFiles, and they'll get automatically run alongside the main tests.
It might be nice to break up the big test file into smaller, more manageable pieces.
Very nice.
Now, the question I have is: How small do you want to make the pieces / tests?
There is something kind of satisfying about the simplicity of having one test per file, similar to how PHP's .phpt "make test" system works.
A structure like this helps somewhat with the "different tests must have different names" requirement, in that you can't have a file with the same name twice in the same directory.
It also makes it easier for people to add tests and send them in (a whole new small file is easier for most folks to work with than a diff), and it's easy to disable tests (just move them into another directory), and it's easy to get a quick overview of what the current tests are (just do an "ls").
Basically in this system $wgParserTestFiles would probably become $wgParserTestDirs, and it would be a directory name rather than a test .txt filename that was added to the global, and every file in a directory would be a set of parserTest tests to be run, and parserTests.txt would probably be split up into 400+ files in maintenance/parserTests/*.txt, and the testitem table would maybe get an extra field (ti_dir) and ti_name might be renamed ti_filename.
Thoughts? Sounds good, or sounds sucky?
All the best, Nick.
Nick Jenkins wrote:
There is something kind of satisfying about the simplicity of having one test per file, similar to how PHP's .phpt "make test" system works.
[snip]
It also makes it easier for people to add tests and send them in (a whole new small file is easier for most folks to work with than a diff), and it's easy to disable tests (just move them into another directory), and it's easy to get a quick overview of what the current tests are (just do an "ls").
Basically in this system $wgParserTestFiles would probably become $wgParserTestDirs, and it would be a directory name rather than a test .txt filename that was added to the global, and every file in a directory would be a set of parserTest tests to be run, and parserTests.txt would probably be split up into 400+ files in maintenance/parserTests/*.txt, and the testitem table would maybe get an extra field (ti_dir) and ti_name might be renamed ti_filename.
Thoughts? Sounds good, or sounds sucky?
Subdirectories might be nice, so those files can be found by the poor sods who have to edit them. ;)
-- brion vibber (brion @ pobox.com)
wikitech-l@lists.wikimedia.org