parserTests is a developer tool to stay informed about potential problems in a release.
Consequently, its customer base is the developer community. I am working on the assumption
that user requirements for changes to it should come from this community (hence the use of
this list for discussions about them).
Originally, I changed parserTests by adding a flipresult option that changed a success to
a failure and a failure to a success. This was universally panned and a requirement for a
know-to-fail status indicated (see emails from Meijssen 7/21 12:19am and 2:56am and Gregor
7/21 4:09am and 9:51am). So, I added this status. In addition, Stephen Bain suggested a
command line option that controls whether known-to-fail test results are accumulated in
the failure or success statistics. So, I added --ktf-to-fail (there is controversy whether
this is the best name. I am open to change it to whatever people want or to get rid of it
if that is the consensus).
Your suggestions are to use the disabled option to turn off those tests known to fail and
to add a --run-disabled command line switch that when set runs the disabled tests. I can
change parserTests to do this is if that is what people want, but I see at least one
issue. You may want to disable specific tests for reasons other than they are known to
fail (e.g., a test exercises functionality undergoing modification and currently not
working). So, you loose the option of separating the known to fail cases from others.
The interactions with --compare and --record are as follows. If we keep the status of
known-to-fail, then should its statistics accumulate with success, failure or neither. The
motivation of the --ktf-to-fail switch is to specify that known-to-fail tests accumulate
with failure statistics. If it is missing, do you want the records to indicate that all of
a sudden you have 14 more successes? If that is acceptable, then the issue is resolved. Or
should known-to-fail results accumulate against neither? In either case, what information
do you want to record and compare against in the testrun and testitem tables? Do you want
to add a column to testrun that indicates whether the --ktf-to-fail or --run-disabled
flags were set? Do you want to add a column to testitem that records known-to-fail status?
Do you want to leave these tables as they are?
In any case, as I stated previously, I can do pretty much whatever the community thinks is
right. I just need a concrete set of user requirements.
--- On Thu, 7/23/09, Brion Vibber <brion(a)wikimedia.org> wrote:
From: Brion Vibber <brion(a)wikimedia.org>
Subject: Re: [Wikitech-l] What to do about --compare and --record. Second request for
comments
To: wikitech-l(a)lists.wikimedia.org
Date: Thursday, July 23, 2009, 4:09 PM
On 07/23/2009 11:00 AM, dan nessett
wrote:
So far no one has responded to my question about how
--ktf-to-fail should interact
with --compare and --record.
Also, at least one commenter has suggested a different name
for --ktf-to-fail. Before I open a bug and attach the
patches, I would like to resolve these issues. Since Brion
suggested this task, would he comment?
Offhand I'm not sure I see a need for a switch
specifically.
Couple thoughts offhand:
* There appears to already be a "disabled" option which can
be added to
test cases. Since this already exists, it doesn't need to
be developed
and could simply be added to the tests we know don't
currently work.
* If there's a desire to run those tests anyway, I'd
probably call the
option --run-disabled. This should be easy to add.
* Not sure there's any need for specific handling w/
compare and record;
we can just record whatever we run.
If on the other hand we want to run and record these tests,
but not
whinge at the user about them, then we'd want another
option on them.
Probably just having another completion state for the
output would do it
(grouping known-to-fail tests separately from others that
fail). I'm not
sure how important that is, though.
-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l