I have modified parserTests to take a "known to fail" switch so those tests that have always failed now pass. It was pretty simple. It only required 3 changes to parserTests.inc and some editing on parserTests.txt. I added an option for each test called flipresult. When this option is specified, the test succeeds when it fails and vice versa.
I have tested the modified parserTest on 1.16a running over a 1.14 schema database. However, I have run into a problem attempting to install the latest trunk revision so I can test against it. Specifically, I added a database called wikitestdb so I would have a production and test wiki. However, when I checked out the latest trunk revision, ran the install script and update.php, and then accessed http://<wiki path>/index.php the installation gets into a infinite redirect loop. When I attempted to debug this (using netbeans 6.7 and Xdebug) the redirect doesn't appear. That is, Main_Page is rendered and displayed. The only difference between the two URLs are the first uses http://<wiki path>/index.php (which redirects to http://<wiki path>/index.php/Main_Page), while the debug session specifies http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDE....
I need some help figuring out what is happening. I imagine using this list for that purpose would be inappropriate. So, if someone would volunteer to help me (email me at the from address in this email), I can get the parserTest changes tested against the newest revision. I can then open a bug (or use an already open bug) and attach the patch and edited parserTests.txt file to it.
Correction: When I tried to create a patch for parserTests.inc and parserTest.txt I discovered the directory I was working from was not under SVN control. So, I created a new working copy of 52088 and when I ran the tests, they reported a databased selection error. So, I cannot claim the tests work for 1.16a under a 1.14 schema. I am now checking out REL1_14 and will run the modified parserTest under it.
--- On Mon, 7/20/09, dan nessett dnessett@yahoo.com wrote:
From: dan nessett dnessett@yahoo.com Subject: [Wikitech-l] "known to fail" switch added to parserTests To: wikitech-l@lists.wikimedia.org Date: Monday, July 20, 2009, 8:09 AM
I have modified parserTests to take a "known to fail" switch so those tests that have always failed now pass. It was pretty simple. It only required 3 changes to parserTests.inc and some editing on parserTests.txt. I added an option for each test called flipresult. When this option is specified, the test succeeds when it fails and vice versa.
I have tested the modified parserTest on 1.16a running over a 1.14 schema database. However, I have run into a problem attempting to install the latest trunk revision so I can test against it. Specifically, I added a database called wikitestdb so I would have a production and test wiki. However, when I checked out the latest trunk revision, ran the install script and update.php, and then accessed http://<wiki path>/index.php the installation gets into a infinite redirect loop. When I attempted to debug this (using netbeans 6.7 and Xdebug) the redirect doesn't appear. That is, Main_Page is rendered and displayed. The only difference between the two URLs are the first uses http://<wiki path>/index.php (which redirects to http://<wiki path>/index.php/Main_Page), while the debug session specifies http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDE....
I need some help figuring out what is happening. I imagine using this list for that purpose would be inappropriate. So, if someone would volunteer to help me (email me at the from address in this email), I can get the parserTest changes tested against the newest revision. I can then open a bug (or use an already open bug) and attach the patch and edited parserTests.txt file to it.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
OK. I think I am now in sync. I checked out 1.14.1 (from the branches arm of r53568) and merged the local changes to parserTests.inc and parserTests.txt. Out of the box, 1.14.1 generates 40 parser test failures. With the code modifications, it generates 24 (i.e., 14 fewer). I have the patches for 1.14.1 and can attach them to an appropriate bug or create a new bug and attach them there.
I also have the patches to r53551, which I can't test because, as I stated previously, my testwiki gets into a redirect loop. No one has stepped forward to help, so I can either provide these for someone else to test or wander around trying to figure out why the redirect loop occurs on my own. One further bit of info on this - when I get to the main page using the debugger, any local link I attempt to navigate to leads me back to the main page, which dies because of the redirect loop. This suggests something is stuck in the redirect logic and everything (including the main page) is redirecting back to the main page.
If someone could let me know how to proceed with the patches, that would help.
--- On Mon, 7/20/09, dan nessett dnessett@yahoo.com wrote:
From: dan nessett dnessett@yahoo.com Subject: Re: [Wikitech-l] "known to fail" switch added to parserTests To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Monday, July 20, 2009, 11:01 AM
Correction: When I tried to create a patch for parserTests.inc and parserTest.txt I discovered the directory I was working from was not under SVN control. So, I created a new working copy of 52088 and when I ran the tests, they reported a databased selection error. So, I cannot claim the tests work for 1.16a under a 1.14 schema. I am now checking out REL1_14 and will run the modified parserTest under it.
--- On Mon, 7/20/09, dan nessett dnessett@yahoo.com wrote:
From: dan nessett dnessett@yahoo.com Subject: [Wikitech-l] "known to fail" switch added to
parserTests
To: wikitech-l@lists.wikimedia.org Date: Monday, July 20, 2009, 8:09 AM
I have modified parserTests to take a "known to fail" switch so those tests that have always failed now
pass. It
was pretty simple. It only required 3 changes to parserTests.inc and some editing on parserTests.txt. I
added
an option for each test called flipresult. When this
option
is specified, the test succeeds when it fails and
vice
versa.
I have tested the modified parserTest on 1.16a running
over
a 1.14 schema database. However, I have run into a
problem
attempting to install the latest trunk revision so I
can
test against it. Specifically, I added a database
called
wikitestdb so I would have a production and test
wiki.
However, when I checked out the latest trunk revision,
ran
the install script and update.php, and then accessed http://<wiki path>/index.php the installation
gets
into a infinite redirect loop. When I attempted to
debug
this (using netbeans 6.7 and Xdebug) the redirect
doesn't
appear. That is, Main_Page is rendered and displayed.
The
only difference between the two URLs are the first
uses
http://<wiki path>/index.php (which redirects
to
http://<wiki path>/index.php/Main_Page), while
the
debug session specifies http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDE....
I need some help figuring out what is happening. I
imagine
using this list for that purpose would be
inappropriate. So,
if someone would volunteer to help me (email me at the
from
address in this email), I can get the parserTest
changes
tested against the newest revision. I can then open a
bug
(or use an already open bug) and attach the patch and
edited
parserTests.txt file to it.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hoi, Suppose that someone fixes a test that has been always failing... one of those "known to fail". It makes no difference right ?? Giving them the status of pass is imho dead wrong because they should not fail in the first place.. now a status of KNOWN TO FAIL makes sense. Thanks, GeardM
2009/7/20 dan nessett dnessett@yahoo.com
I have modified parserTests to take a "known to fail" switch so those tests that have always failed now pass. It was pretty simple. It only required 3 changes to parserTests.inc and some editing on parserTests.txt. I added an option for each test called flipresult. When this option is specified, the test succeeds when it fails and vice versa.
I have tested the modified parserTest on 1.16a running over a 1.14 schema database. However, I have run into a problem attempting to install the latest trunk revision so I can test against it. Specifically, I added a database called wikitestdb so I would have a production and test wiki. However, when I checked out the latest trunk revision, ran the install script and update.php, and then accessed http://<wiki path>/index.php the installation gets into a infinite redirect loop. When I attempted to debug this (using netbeans 6.7 and Xdebug) the redirect doesn't appear. That is, Main_Page is rendered and displayed. The only difference between the two URLs are the first uses http://<wiki path>/index.php (which redirects to http://<wiki path>/index.php/Main_Page), while the debug session specifies
http://localhost/MediawikiTest/Latest%20Trunk%20Version/phase3/index.php?XDE... .
I need some help figuring out what is happening. I imagine using this list for that purpose would be inappropriate. So, if someone would volunteer to help me (email me at the from address in this email), I can get the parserTest changes tested against the newest revision. I can then open a bug (or use an already open bug) and attach the patch and edited parserTests.txt file to it.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Jul 21, 2009 at 7:19 AM, Gerard Meijssengerard.meijssen@gmail.com wrote:
Suppose that someone fixes a test that has been always failing... one of those "known to fail". It makes no difference right ??
Difference in what sense? It means we have one less failing test reported, presumably.
Giving them the status of pass is imho dead wrong because they should not fail in the first place.. now a status of KNOWN TO FAIL makes sense.
The known-failing tests have never passed. They're a wishlist. None of them are likely to be fixed in the foreseeable future. I'd be fine with just removing them, but Brion has been against it in the past.
Hoi, There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of "known to fail" makes sense, just changing them to pass makes it necessary to add a "citation needed" because it is just not true. Thanks, GerardM
2009/7/21 Aryeh Gregor <Simetrical+wikilist@gmail.comSimetrical%2Bwikilist@gmail.com
On Tue, Jul 21, 2009 at 7:19 AM, Gerard Meijssengerard.meijssen@gmail.com wrote:
Suppose that someone fixes a test that has been always failing... one of those "known to fail". It makes no difference right ??
Difference in what sense? It means we have one less failing test reported, presumably.
Giving them the status of pass is imho dead wrong because they should not fail in the
first
place.. now a status of KNOWN TO FAIL makes sense.
The known-failing tests have never passed. They're a wishlist. None of them are likely to be fixed in the foreseeable future. I'd be fine with just removing them, but Brion has been against it in the past.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijssen@gmail.com wrote:
There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of "known to fail" makes sense, just changing them to pass makes it necessary to add a "citation needed" because it is just not true.
Nobody's changing them to pass.
Aryeh Gregor wrote:
On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijssen@gmail.com wrote:
There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of "known to fail" makes sense, just changing them to pass makes it necessary to add a "citation needed" because it is just not true.
Nobody's changing them to pass.
I can understand where Gerard got the impression:
"I have modified parserTests to take a "known to fail" switch so those tests that have always failed now pass." - dan nessett 2009-07-20 16:09
KTC
On Tue, Jul 21, 2009 at 7:43 AM, Kwan Ting Chanktc@ktchan.info wrote:
Aryeh Gregor wrote:
On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijssen@gmail.com wrote:
There is no point having a perfect score when it is actually a lie. It seems to me that Brion is against the removal of these tests because he wants them to pass. Having a third state of "known to fail" makes sense, just changing them to pass makes it necessary to add a "citation needed" because it is just not true.
Nobody's changing them to pass.
I can understand where Gerard got the impression:
"I have modified parserTests to take a "known to fail" switch so those tests that have always failed now pass." - dan nessett 2009-07-20 16:09
KTC
-- Experience is a good school but the fees are high. - Heinrich Heine
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
That being said, a patch against 1.14.x is of no use to anyone.
-Chad
The change I made was to add a "flipresult" option that simply turns a success into a failure and a failure into a success. This is what I understood I was asked to do. On the plus side, this approach also allows the addition of parser tests that are supposed to fail (not just have always failed). On the negative side it does hide problems that perhaps should remain in the open.
I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either.
Would someone tell me where the redirect from index.php to index.php/Main_Page occurs in the page processing flow?
--- On Tue, 7/21/09, Kwan Ting Chan ktc@ktchan.info wrote:
From: Kwan Ting Chan ktc@ktchan.info Subject: Re: [Wikitech-l] "known to fail" switch added to parserTests To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Tuesday, July 21, 2009, 4:43 AM Aryeh Gregor wrote:
On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijssen@gmail.com
wrote:
There is no point having a perfect score when it
is actually a lie. It seems
to me that Brion is against the removal of these
tests because he wants them
to pass. Having a third state of "known to fail"
makes sense, just changing
them to pass makes it necessary to add a "citation
needed" because it is
just not true.
Nobody's changing them to pass.
I can understand where Gerard got the impression:
"I have modified parserTests to take a "known to fail" switch so those tests that have always failed now pass." - dan nessett 2009-07-20 16:09
KTC
-- Experience is a good school but the fees are high. - Heinrich Heine
-----Inline Attachment Follows-----
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Jul 22, 2009 at 12:45 AM, dan nessettdnessett@yahoo.com wrote:
I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either.
Having a third possible result would be more informative.
You could report two scores, one including known-to-fail tests and one excluding them. Reporting both or just one of these scores could be configurable by a command line option.
On Tue, Jul 21, 2009 at 10:45 AM, dan nessettdnessett@yahoo.com wrote:
The change I made was to add a "flipresult" option that simply turns a success into a failure and a failure into a success. This is what I understood I was asked to do. On the plus side, this approach also allows the addition of parser tests that are supposed to fail (not just have always failed). On the negative side it does hide problems that perhaps should remain in the open.
This isn't a good idea, no. The important thing is if someone runs parserTests.php, they should be able to easily tell whether there are any regressions in their working copy. But if we're going to keep the known-to-fail tests at all, it doesn't make a lot of sense to report them as passing when they're actually failing . . . if we do that we may as well just drop them.
I just looked at the code and it shouldn't be hard to add a knowntofail option that acts like flipresult and then add a new category of test result that specifies how many known to fail results occurred. However, one issue is whether known to fail should count against success/failure (when computing the percentage of tests that failed) or whether these results should not count toward either.
It should be made clear that some tests are failing, but that they are not regressions.
Would someone tell me where the redirect from index.php to index.php/Main_Page occurs in the page processing flow?
I don't know offhand.
On Tue, Jul 21, 2009 at 10:51 AM, Aryeh Gregor < Simetrical+wikilist@gmail.com Simetrical%2Bwikilist@gmail.com> wrote:
But if we're going to keep the known-to-fail tests at all, it doesn't make a lot of sense to report them as passing when they're actually failing . . . if we do that we may as well just drop them.
The tests have never passed - they should be commented out for usability reasons. And ideally there would be a post-commit hook that runs the parser tests and e-mails the committer letting them know they have just broken the software!
On Tue, Jul 21, 2009 at 12:57 PM, BrianBrian.Mingus@colorado.edu wrote:
The tests have never passed - they should be commented out for usability reasons.
Well, I have no objections, but apparently that's not acceptable. An "expected fail" flag would be about as usable.
And ideally there would be a post-commit hook that runs the parser tests and e-mails the committer letting them know they have just broken the software!
Yes, that would be nice.
Not sure the post-commit hook running the parser is a good idea. The software could have been broken by a previous committer. From that point on parserTests will report errors until the problem is fixed, so committers will just learn to ignore the message.
--- On Tue, 7/21/09, Brian Brian.Mingus@colorado.edu wrote:
From: Brian Brian.Mingus@colorado.edu Subject: Re: [Wikitech-l] "known to fail" switch added to parserTests To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Tuesday, July 21, 2009, 9:57 AM On Tue, Jul 21, 2009 at 10:51 AM, Aryeh Gregor < Simetrical+wikilist@gmail.com Simetrical%2Bwikilist@gmail.com> wrote:
... And ideally there would be a post-commit hook that
runs the parser tests and e-mails the committer letting them know they have just broken the software! _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Jul 21, 2009 at 12:05 PM, dan nessett dnessett@yahoo.com wrote:
Not sure the post-commit hook running the parser is a good idea. The software could have been broken by a previous committer. From that point on parserTests will report errors until the problem is fixed, so committers will just learn to ignore the message.
Right, well, a pre-commit hook that rejects all commits which break the software. Or a memory of what commits broke which tests and a conditional.
Better ideas. Another possibility is every 24hrs to run parser tests (and any other regression tests that might exist) against all revisions committed into trunk since the last run. Post the results and keep track of the number of bugs each committer has introduced into the code base for the past running 6 month period. Post the names of committers and the number of bugs they have introduced on a "hall of shame" page ordering the list by number of bugs.
Sometimes social pressure can be a very effective behavior modifier.
--- On Tue, 7/21/09, Brian Brian.Mingus@colorado.edu wrote:
From: Brian Brian.Mingus@colorado.edu Subject: Re: [Wikitech-l] "known to fail" switch added to parserTests To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Tuesday, July 21, 2009, 11:10 AM
Right, well, a pre-commit hook that rejects all commits which break the software. Or a memory of what commits broke which tests and a conditional. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Jul 21, 2009 at 6:05 PM, dan nessettdnessett@yahoo.com wrote:
Not sure the post-commit hook running the parser is a good idea. The software could have been broken by a previous committer. From that point on parserTests will report errors until the problem is fixed, so committers will just learn to ignore the message.
It can use --record and --compare.
Hoi, That is exactly the problem. You report that they pass and in reality they still fail. Someone should change them to pass ie fix the software. Thanks, GerardM
2009/7/21 Aryeh Gregor <Simetrical+wikilist@gmail.comSimetrical%2Bwikilist@gmail.com
On Tue, Jul 21, 2009 at 9:56 AM, Gerard Meijssengerard.meijssen@gmail.com wrote:
There is no point having a perfect score when it is actually a lie. It
seems
to me that Brion is against the removal of these tests because he wants
them
to pass. Having a third state of "known to fail" makes sense, just
changing
them to pass makes it necessary to add a "citation needed" because it is just not true.
Nobody's changing them to pass.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Jul 21, 2009 at 3:21 PM, Gerard Meijssengerard.meijssen@gmail.com wrote:
Hoi, That is exactly the problem. You report that they pass and in reality they still fail. Someone should change them to pass ie fix the software.
I think the appropriate English reply in this context would be "No s**t, Sherlock"...
wikitech-l@lists.wikimedia.org