"dan nessett" <dnessett(a)yahoo.com> wrote in message
news:102343.8273.qm@web32502.mail.mud.yahoo.com...
So far no one has responded to Brion's comments about how parserTests
should be modified.
You are the customers. What do you want?
Should I just modify parserTests.txt to disable those tests that always
have failed? Should I
do that and add an option to run disabled tests?
Or should I retain the known-to-fail status? If so, should I retain the
option that
known-to-fail results accumulate as fails? If so, what should the option
be called? Right
now it is --ktf-to-fail. There is one proposal to change it
to --with-known-to-fail. Any
others? Which do people prefer?
Ignoring the question of whether the test should be there or not (and
assuming they are left in), here's how I read the situation:
* Most tests are expected to pass. If they fail, this is bad, if they pass
this is good. If the status changes, this should be reported.
* Some tests are expected to fail. If they fail this is OK, if they pass
this is excellent. If the status changes, this should be reported.
Therefore there should be an 'ExpectedResult' field in the test definition,
which is either 'pass' or 'fail', and which defaults to 'pass' if
not
present.
The output from the parser tests should show the results for all four
possible outcomes (possibly omitting items with zero elements). All
compare/record functionality should be logging these results so that we get
alerts about known-to-fail tests which start to pass.
I don't see a need for any extra command-line args if implemented as
described above.
- Mark Clements (HappyDog)