It appears there is no clear consensus in the developer community how to resolve the problem of the parserTests known-to-fail tests. Consequently, the most useful thing I can do is provide the patches I have developed and explain how they can be used to implement one of a number of options. These are neither the only options nor is one of them necessarily the best. They simply correspond to suggestions made on the wikitech-1 email list (the second option is additional). There is, of course, the possibility of developing an option not specified here and implementing it.

[Mark Clements added another option this morning - 7/27/09 - that requires additional development. If that is what people want, I (or others) can do the necessary implementation. However, I thought it best to get out what I have done so far in case it meets consensus needs.]

I have attached the patches to bug 19957 (I couldn't find an appropriate existing bug for this issue, so I created one. Also, I attached a tar file with 7 patches. When I looked at the bug after submitting it, it wasn't clear the tar file made it. Any advice?). These patches can be used to implement the options specified below. Details of the patches are found in text I have added to the bug report.

+ Install none of these patches and do nothing else:

+ Pros: parserTests behavior remains as currently defined. + Cons: parserTests behavior remains as currently defined.

+ Install none of these patches and define successful output as that which the Parser currently returns (this would require changing parserTests.txt using a unprovided patch):

+ Pros: parserTests is brought into compliance with the test set. + Cons: It is generally bad practice to define success without understanding the logic behind it.

+ Install none of these patches, file bugs against the 14 failing tests and then fix the bugs:

+ Pros: parserTests is brought into compliance with the test set. + Cons: It has been suggested that the parserTests logic is sufficiently complex that it may be difficult to modify it to pass the known to fail tests.

+ File bugs against the 14 failing tests and patch parserTests.txt to comment out the known to fail tests:

+ Pros: Testing the Parser using parserTests does not return confusing test results. Developers can concentrate on those tests that fail due to the addition of new functionality or intefering bug fixes. + Cons: Since the known to fail tests no longer report their presence, it may be easy for the community to forget they are still a problem.

+ Patch parserTests.txt so the known to fail tests are disabled. Optionally file bugs against these tests:

+ Pros: Testing the Parser using parserTests does not return confusing test results. Developers can concentrate on those tests that fail due to the addition of new functionality or intefering bug fixes. + Cons: There are other disabled tests in parserTests.txt. These could become confused with the known to fail tests.

+ Patch parserTests|.inc|.php to provide a --run-disabled option:

+ Pros: This would allow disabled known to fail tests to run without editing parserTests.txt + Cons: Specifciation of this option runs all disabled tests, not just those known to fail. Test results using this option might confuse developers.

+ Patch parserTests|.inc|.txt to provide a knowntofail option:

+ Pros: parserTests output and summary statistics indicate which tests succeeded, failed and returned known-to-fail status. + Cons: Creates a new test status for what should be a temporary problem. This is a questionable software architecture decision.

+ Patch parserTests.php to support a ktf-in-fail option:

+ Pros: those of the previous option. Adds the ability to accumulate ktf failure status in failure statistics. + Cons: those of the previous option. Raises the question of what is the proper statistics strategy when the known-to-fail option is missing (on runs with this option missing, known-to-fail tests accumulate against success statistics).

The provided patches do not address whether the 'testrun' and 'testitem' tables should be modifed to note the specification of statistics changing application options (i.e., either --ktf-in-fail or --run-disabled) or the use of the knowntofail option for a particular test. These tables are not modified by the patches.