I have added an application option ktf-to-fail that when specified accumulates tests with known-to-fail status as if they failed. When this option is missing, failure statistics do not include known-to-fail results and there is a summary at the end of parserTests that specifies how many known-to-fail tests were run (unless that number is zero). I also have modified parserTests to indicate the known-to-fail status when that option is specified.
But, there is still an issue. How should the per-test known-to-fail option interact with the compare and record application options? Should parserTests be modified to record and compare known-to-fail results? Or, should these results be silent for recording purposes and treated as failures only if the ktf-to-fail application options is specified?
I assume "ktf" is short for Known To Fail? You've got a bit of RAS syndrome going on there... :-D
--HM
"dan nessett" dnessett@yahoo.com wrote in message news:842292.50373.qm@web32504.mail.mud.yahoo.com...
I have added an application option ktf-to-fail that when specified accumulates tests with known-to-fail status as if they failed. When this option is missing, failure statistics do not include known-to-fail results and there is a summary at the end of parserTests that specifies how many known-to-fail tests were run (unless that number is zero). I also have modified parserTests to indicate the known-to-fail status when that option is specified.
But, there is still an issue. How should the per-test known-to-fail option interact with the compare and record application options? Should parserTests be modified to record and compare known-to-fail results? Or, should these results be silent for recording purposes and treated as failures only if the ktf-to-fail application options is specified?
Right. The option is a bit cryptic. I first thought of knowntofail-to-failures, but that was way too long.
--- On Wed, 7/22/09, Happy-melon happy-melon@live.com wrote:
From: Happy-melon happy-melon@live.com Subject: Re: [Wikitech-l] Known to fail interactions with compare and record To: wikitech-l@lists.wikimedia.org Date: Wednesday, July 22, 2009, 3:44 PM I assume "ktf" is short for Known To Fail? You've got a bit of RAS syndrome going on there... :-D
--HM
"dan nessett" dnessett@yahoo.com wrote in message news:842292.50373.qm@web32504.mail.mud.yahoo.com...
I have added an application option ktf-to-fail that
when specified
accumulates tests with known-to-fail status as if they
failed. When this
option is missing, failure statistics do not include
known-to-fail results
and there is a summary at the end of parserTests that
specifies how many
known-to-fail tests were run (unless that number is
zero). I also have
modified parserTests to indicate the known-to-fail
status when that option
is specified.
But, there is still an issue. How should the per-test
known-to-fail option
interact with the compare and record application
options? Should
parserTests be modified to record and compare
known-to-fail results? Or,
should these results be silent for recording purposes
and treated as
failures only if the ktf-to-fail application options
is specified?
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Why not just call it --with-known-to-fail? Easy.
-Chad
On Jul 22, 2009 7:11 PM, "dan nessett" dnessett@yahoo.com wrote:
Right. The option is a bit cryptic. I first thought of knowntofail-to-failures, but that was way too long.
--- On Wed, 7/22/09, Happy-melon happy-melon@live.com wrote:
From: Happy-melon happy-melon@live.com Subject: Re: [Wikitech-l] Known to fail interactions with compare and
record
To: wikitech-l@lists.wikimedia.org Date: Wednesday, July 22, 2009, 3:44 PM
I assume "ktf" is short for Known To > Fail? You've got a bit of RAS
syndrome > going on there.....
If anyone can come up with a better option name, I would be happy to replace ktf-to-fail. I generally don't like cryptic abbreviations. However, "with-known-to-fail" doesn't really get at the underlying meaning of the option. It specifies that known-to-fail test results are accumulated as failures for the purpose of parserTest statistics.
--- On Wed, 7/22/09, Chad innocentkiller@gmail.com wrote:
From: Chad innocentkiller@gmail.com Subject: Re: [Wikitech-l] Known to fail interactions with compare and record To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Wednesday, July 22, 2009, 4:15 PM Why not just call it --with-known-to-fail? Easy.
-Chad
On Jul 22, 2009 7:11 PM, "dan nessett" dnessett@yahoo.com wrote:
Right. The option is a bit cryptic. I first thought of knowntofail-to-failures, but that was way too long.
--- On Wed, 7/22/09, Happy-melon happy-melon@live.com wrote:
From: Happy-melon happy-melon@live.com Subject: Re: [Wikitech-l] Known to fail interactions
with compare and record
To: wikitech-l@lists.wikimedia.org Date: Wednesday, July 22, 2009, 3:44 PM
I assume "ktf" is short for Known To > Fail?
You've got a bit of RAS syndrome > going on there..... _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Which is exactly what my param means. Its expected that failures will be reported, and --with-known-to-fail would indicate that known failures will be added.
--knowntofail-to-fail and --ktf-to-fail certainly aren't any clearer.
-Chad
On Jul 22, 2009 7:22 PM, "dan nessett" dnessett@yahoo.com wrote:
If anyone can come up with a better option name, I would be happy to replace ktf-to-fail. I generally don't like cryptic abbreviations. However, "with-known-to-fail" doesn't really get at the underlying meaning of the option. It specifies that known-to-fail test results are accumulated as failures for the purpose of parserTest statistics.
--- On Wed, 7/22/09, Chad innocentkiller@gmail.com wrote:
From: Chad innocentkiller@gmail.com
Subject: Re: [Wikitech-l] Known to fail interactions with compare and
record
To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Wednesday, July 22, 2009, 4:15 PM
Why not just call it > --with-known-to-fail? Easy. > > -Chad > > On Jul
22, 2009 7:11 PM, "dan n...
_______________________________________________ > Wikitech-l mailing list Wikitech-l@lists.wikim...
Well, it isn't all that clear to me, but I really don't care. I'll change it to whatever people want. "Call me anything you like, but don't call me late for dinner."
Can someone tell me how the --fuzz option is supposed to behave? I am cross-testing the new parserTests parameter in conjunction with its other parameters. I have tested --quick and --quiet. They seem to work fine with ktf-to-fail. When I test --fuzz, parserTests seems to go on walkabout in the Great Australian desert periodically spewing out stuff like:
100: 100/100 (mem: 36%) 200: 200/200 (mem: 37%) 300: 300/300 (mem: 37%) 400: 400/400 (mem: 37%) 500: 500/500 (mem: 38%) 600: 600/600 (mem: 38%) ....
Is this expected behavior? Is parserTests supposed to finish when you use --fuzz or is this some kind of stress test that the never finishes?
--- On Wed, 7/22/09, Chad innocentkiller@gmail.com wrote:
From: Chad innocentkiller@gmail.com Subject: Re: [Wikitech-l] Known to fail interactions with compare and record To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Wednesday, July 22, 2009, 4:25 PM Which is exactly what my param means. Its expected that failures will be reported, and --with-known-to-fail would indicate that known failures will be added.
--knowntofail-to-fail and --ktf-to-fail certainly aren't any clearer.
-Chad
On Jul 22, 2009 7:22 PM, "dan nessett" dnessett@yahoo.com wrote:
If anyone can come up with a better option name, I would be happy to replace ktf-to-fail. I generally don't like cryptic abbreviations. However, "with-known-to-fail" doesn't really get at the underlying meaning of the option. It specifies that known-to-fail test results are accumulated as failures for the purpose of parserTest statistics.
--- On Wed, 7/22/09, Chad innocentkiller@gmail.com wrote:
From: Chad innocentkiller@gmail.com
Subject: Re: [Wikitech-l] Known to fail interactions
with compare and record
To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Wednesday, July 22, 2009, 4:15 PM
Why not just call it > --with-known-to-fail? Easy.
-Chad > > On Jul
22, 2009 7:11 PM, "dan n...
_______________________________________________ >
Wikitech-l mailing list
Wikitech-l@lists.wikim...
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
dan nessett wrote:
Well, it isn't all that clear to me, but I really don't care. I'll change it to whatever people want. "Call me anything you like, but don't call me late for dinner."
Can someone tell me how the --fuzz option is supposed to behave? I am cross-testing the new parserTests parameter in conjunction with its other parameters. I have tested --quick and --quiet. They seem to work fine with ktf-to-fail. When I test --fuzz, parserTests seems to go on walkabout in the Great Australian desert periodically spewing out stuff like:
100: 100/100 (mem: 36%) 200: 200/200 (mem: 37%) 300: 300/300 (mem: 37%) 400: 400/400 (mem: 37%) 500: 500/500 (mem: 38%) 600: 600/600 (mem: 38%) ....
Is this expected behavior? Is parserTests supposed to finish when you use --fuzz or is this some kind of stress test that the never finishes?
It runs forever, unless it runs out of memory or hits a fatal PHP error. It's not a stress test, it's a fuzz test, hence the name. It logs exceptions generated by the parser for random input.
Maybe if there's an undocumented option that you don't understand, you should leave it alone. Otherwise some day your wiki will end up with all its articles deleted, or with all the text converted to ISO-2022-JP or something.
-- Tim Starling
Thanks. Just to clarify, I am not changing --fuzz. I am testing --ktf-to-fail in conjunction with other parserTests options to ensure there is no interference. The chances of such interference is very small, but since I have been preaching the importance of regression testing, I thought I should eat my own dog-food.
--- On Wed, 7/22/09, Tim Starling tstarling@wikimedia.org wrote:
From: Tim Starling tstarling@wikimedia.org Subject: Re: [Wikitech-l] Known to fail interactions with compare and record To: wikitech-l@lists.wikimedia.org Date: Wednesday, July 22, 2009, 10:17 PM dan nessett wrote:
Well, it isn't all that clear to me, but I really
don't care. I'll
change it to whatever people want. "Call me anything
you like, but
don't call me late for dinner."
Can someone tell me how the --fuzz option is supposed
to behave? I
am cross-testing the new parserTests parameter in
conjunction with
its other parameters. I have tested --quick and
--quiet. They seem
to work fine with ktf-to-fail. When I test --fuzz,
parserTests
seems to go on walkabout in the Great Australian
desert
periodically spewing out stuff like:
100: 100/100 (mem: 36%) 200: 200/200 (mem: 37%) 300:
300/300 (mem:
37%) 400: 400/400 (mem: 37%) 500: 500/500 (mem: 38%)
600: 600/600
(mem: 38%) ....
Is this expected behavior? Is parserTests supposed to
finish when
you use --fuzz or is this some kind of stress test
that the never
finishes?
It runs forever, unless it runs out of memory or hits a fatal PHP error. It's not a stress test, it's a fuzz test, hence the name. It logs exceptions generated by the parser for random input.
Maybe if there's an undocumented option that you don't understand, you should leave it alone. Otherwise some day your wiki will end up with all its articles deleted, or with all the text converted to ISO-2022-JP or something.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org