parserTests is a developer tool to stay informed about potential problems in a release. Consequently, its customer base is the developer community. I am working on the assumption that user requirements for changes to it should come from this community (hence the use of this list for discussions about them).
Originally, I changed parserTests by adding a flipresult option that changed a success to a failure and a failure to a success. This was universally panned and a requirement for a know-to-fail status indicated (see emails from Meijssen 7/21 12:19am and 2:56am and Gregor 7/21 4:09am and 9:51am). So, I added this status. In addition, Stephen Bain suggested a command line option that controls whether known-to-fail test results are accumulated in the failure or success statistics. So, I added --ktf-to-fail (there is controversy whether this is the best name. I am open to change it to whatever people want or to get rid of it if that is the consensus).
Your suggestions are to use the disabled option to turn off those tests known to fail and to add a --run-disabled command line switch that when set runs the disabled tests. I can change parserTests to do this is if that is what people want, but I see at least one issue. You may want to disable specific tests for reasons other than they are known to fail (e.g., a test exercises functionality undergoing modification and currently not working). So, you loose the option of separating the known to fail cases from others.
The interactions with --compare and --record are as follows. If we keep the status of known-to-fail, then should its statistics accumulate with success, failure or neither. The motivation of the --ktf-to-fail switch is to specify that known-to-fail tests accumulate with failure statistics. If it is missing, do you want the records to indicate that all of a sudden you have 14 more successes? If that is acceptable, then the issue is resolved. Or should known-to-fail results accumulate against neither? In either case, what information do you want to record and compare against in the testrun and testitem tables? Do you want to add a column to testrun that indicates whether the --ktf-to-fail or --run-disabled flags were set? Do you want to add a column to testitem that records known-to-fail status? Do you want to leave these tables as they are?
In any case, as I stated previously, I can do pretty much whatever the community thinks is right. I just need a concrete set of user requirements.
--- On Thu, 7/23/09, Brion Vibber brion@wikimedia.org wrote:
From: Brion Vibber brion@wikimedia.org Subject: Re: [Wikitech-l] What to do about --compare and --record. Second request for comments To: wikitech-l@lists.wikimedia.org Date: Thursday, July 23, 2009, 4:09 PM On 07/23/2009 11:00 AM, dan nessett wrote:
So far no one has responded to my question about how
--ktf-to-fail should interact with --compare and --record. Also, at least one commenter has suggested a different name for --ktf-to-fail. Before I open a bug and attach the patches, I would like to resolve these issues. Since Brion suggested this task, would he comment?
Offhand I'm not sure I see a need for a switch specifically.
Couple thoughts offhand:
- There appears to already be a "disabled" option which can
be added to test cases. Since this already exists, it doesn't need to be developed and could simply be added to the tests we know don't currently work.
- If there's a desire to run those tests anyway, I'd
probably call the option --run-disabled. This should be easy to add.
- Not sure there's any need for specific handling w/
compare and record; we can just record whatever we run.
If on the other hand we want to run and record these tests, but not whinge at the user about them, then we'd want another option on them. Probably just having another completion state for the output would do it (grouping known-to-fail tests separately from others that fail). I'm not sure how important that is, though.
-- brion
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org