So far no one has responded to Brion's comments about how parserTests should be modified. You are the customers. What do you want?
Should I just modify parserTests.txt to disable those tests that always have failed? Should I do that and add an option to run disabled tests?
Or should I retain the known-to-fail status? If so, should I retain the option that known-to-fail results accumulate as fails? If so, what should the option be called? Right now it is --ktf-to-fail. There is one proposal to change it to --with-known-to-fail. Any others? Which do people prefer?
Should we note anything new in the testrun and testitem tables? Specifically, should we add a column to testrun that records whether --ktf-to-fail (or whatever we call it) or --run-disabled was set (obviously the table would be modified only to note one of these depending on which implementation option people choose). If we go with the known-to-fail status, should we add a column to testitem that indicates that a test returned a known-to-fail status? Should we leave these tables alone?
I do not have enough experience to determine which of these are the best options. You do, so you need to choose. [Of course, you can remain silent, which I will interpret to mean you really don't care]
On Fri, Jul 24, 2009 at 10:05 AM, dan nessett dnessett@yahoo.com wrote:
So far no one has responded to Brion's comments about how parserTests should be modified. You are the customers. What do you want?
My two cents: Remove the chronically failing tests and file them as bugs in bugzilla. As each bug is fixed, its corresponding parser test can be put back in.
"Remember the dot" rememberthedot@gmail.com wrote in message news:17aa57b60907241739l5b0822a8k6b79841a5cbcc9ca@mail.gmail.com...
My two cents: Remove the chronically failing tests and file them as bugs in bugzilla. As each bug is fixed, its corresponding parser test can be put back in.
Second that. The correct way to record enhancement requests is using bugzilla, not in live code.
--HM
"dan nessett" dnessett@yahoo.com wrote in message news:102343.8273.qm@web32502.mail.mud.yahoo.com...
So far no one has responded to Brion's comments about how parserTests should be modified. You are the customers. What do you want?
Should I just modify parserTests.txt to disable those tests that always have failed? Should I do that and add an option to run disabled tests?
Or should I retain the known-to-fail status? If so, should I retain the option that known-to-fail results accumulate as fails? If so, what should the option be called? Right now it is --ktf-to-fail. There is one proposal to change it to --with-known-to-fail. Any others? Which do people prefer?
Ignoring the question of whether the test should be there or not (and assuming they are left in), here's how I read the situation:
* Most tests are expected to pass. If they fail, this is bad, if they pass this is good. If the status changes, this should be reported. * Some tests are expected to fail. If they fail this is OK, if they pass this is excellent. If the status changes, this should be reported.
Therefore there should be an 'ExpectedResult' field in the test definition, which is either 'pass' or 'fail', and which defaults to 'pass' if not present.
The output from the parser tests should show the results for all four possible outcomes (possibly omitting items with zero elements). All compare/record functionality should be logging these results so that we get alerts about known-to-fail tests which start to pass.
I don't see a need for any extra command-line args if implemented as described above.
- Mark Clements (HappyDog)
wikitech-l@lists.wikimedia.org