I decided to investigate how well parserTests exercises the MW code. So, I threw together a couple of MacGyver tools that use xdebug's code coverage capability and analyzed the results. The results are very, very preliminary, but I thought I would get them out so others can look them over. In the next couple of days I hope to post more detailed results and the tools themselves on the Mediawiki wiki. (If someone could tell me the appropriate page to use that would be useful. Otherwise, I will just create a page in my own namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of lines in those files: 85606 Lines covered: 59489 Lines not covered: 26117 Percentage covered: 0.694916244188
So, parserTests is getting (at best) about 70% code coverage. This is better than I expected, but still it means parserTests does not test 26117 lines of code. What I mean by "at best" is xdebug just notes whether a line of code is visited. It doesn't do any logic analysis on which branches are taken. Furthermore, parserTests may not visit some files that are critical to the operation of the MW software. Obviously, xdebug can only gather statistics on visited files.
I want to emphasize that there may be errors in these results due to bad assumptions on my part or bad coding. However, it is a place to start.
I failed to mention that xdebug ignores non-executable lines of code. So, the statistics are for executable lines of code and do not include lines like comments (in either the covered or uncovered counts).
--- On Wed, 7/29/09, dan nessett dnessett@yahoo.com wrote:
From: dan nessett dnessett@yahoo.com Subject: [Wikitech-l] parserTests code coverage statistics To: wikitech-l@lists.wikimedia.org Date: Wednesday, July 29, 2009, 4:36 PM
I decided to investigate how well parserTests exercises the MW code. So, I threw together a couple of MacGyver tools that use xdebug's code coverage capability and analyzed the results. The results are very, very preliminary, but I thought I would get them out so others can look them over. In the next couple of days I hope to post more detailed results and the tools themselves on the Mediawiki wiki. (If someone could tell me the appropriate page to use that would be useful. Otherwise, I will just create a page in my own namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of lines in those files: 85606 Lines covered: 59489 Lines not covered: 26117 Percentage covered: 0.694916244188
So, parserTests is getting (at best) about 70% code coverage. This is better than I expected, but still it means parserTests does not test 26117 lines of code. What I mean by "at best" is xdebug just notes whether a line of code is visited. It doesn't do any logic analysis on which branches are taken. Furthermore, parserTests may not visit some files that are critical to the operation of the MW software. Obviously, xdebug can only gather statistics on visited files.
I want to emphasize that there may be errors in these results due to bad assumptions on my part or bad coding. However, it is a place to start.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I suspect that much of this has to do with the way Parser.php is eleven thousand lines of programmatic sewage, and the way the ParserTests infrastructure requires a lot of the rest of MediaWiki to be initialised in order to run the tests. As long as the rest of the system 'works' well enough to allow the parser to parse, ParserTests is happy: I wouldn't be confident to say that it "tests" anything outside of Parser.php; as you say, it only marks lines 'visited'.
--HM
"dan nessett" dnessett@yahoo.com wrote in message news:922893.25678.qm@web32507.mail.mud.yahoo.com...
I failed to mention that xdebug ignores non-executable lines of code. So, the statistics are for executable lines of code and do not include lines like comments (in either the covered or uncovered counts).
--- On Wed, 7/29/09, dan nessett dnessett@yahoo.com wrote:
From: dan nessett dnessett@yahoo.com Subject: [Wikitech-l] parserTests code coverage statistics To: wikitech-l@lists.wikimedia.org Date: Wednesday, July 29, 2009, 4:36 PM
I decided to investigate how well parserTests exercises the MW code. So, I threw together a couple of MacGyver tools that use xdebug's code coverage capability and analyzed the results. The results are very, very preliminary, but I thought I would get them out so others can look them over. In the next couple of days I hope to post more detailed results and the tools themselves on the Mediawiki wiki. (If someone could tell me the appropriate page to use that would be useful. Otherwise, I will just create a page in my own namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of lines in those files: 85606 Lines covered: 59489 Lines not covered: 26117 Percentage covered: 0.694916244188
So, parserTests is getting (at best) about 70% code coverage. This is better than I expected, but still it means parserTests does not test 26117 lines of code. What I mean by "at best" is xdebug just notes whether a line of code is visited. It doesn't do any logic analysis on which branches are taken. Furthermore, parserTests may not visit some files that are critical to the operation of the MW software. Obviously, xdebug can only gather statistics on visited files.
I want to emphasize that there may be errors in these results due to bad assumptions on my part or bad coding. However, it is a place to start.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
=
I have no argument with your points. However, figuring out the code coverage of parserTests is low hanging fruit, i.e., relatively easy to determine and at least somewhat valuable. By knowing which files are touched by parserTests and how much of their code is covered is a first step in figuring out what needs to be done (e.g., identifying those files that parserTests doesn't even visit). After all, parserTests is all we have at the moment.
--- On Thu, 7/30/09, Happy-melon happy-melon@live.com wrote:
From: Happy-melon happy-melon@live.com Subject: Re: [Wikitech-l] parserTests code coverage statistics To: wikitech-l@lists.wikimedia.org Date: Thursday, July 30, 2009, 4:26 AM I suspect that much of this has to do with the way Parser.php is eleven thousand lines of programmatic sewage, and the way the ParserTests infrastructure requires a lot of the rest of MediaWiki to be initialised in order to run the tests. As long as the rest of the system 'works' well enough to allow the parser to parse, ParserTests is happy: I wouldn't be confident to say that it "tests" anything outside of Parser.php; as you say, it only marks lines 'visited'.
--HM
"dan nessett" dnessett@yahoo.com wrote in message news:922893.25678.qm@web32507.mail.mud.yahoo.com...
I failed to mention that xdebug ignores non-executable
lines of code. So,
the statistics are for executable lines of code and do
not include lines
like comments (in either the covered or uncovered
counts).
--- On Wed, 7/29/09, dan nessett dnessett@yahoo.com
wrote:
From: dan nessett dnessett@yahoo.com Subject: [Wikitech-l] parserTests code coverage
statistics
To: wikitech-l@lists.wikimedia.org Date: Wednesday, July 29, 2009, 4:36 PM
I decided to investigate how well parserTests
exercises the
MW code. So, I threw together a couple of MacGyver
tools
that use xdebug's code coverage capability and
analyzed the
results. The results are very, very preliminary,
but I
thought I would get them out so others can look
them over.
In the next couple of days I hope to post more
detailed
results and the tools themselves on the Mediawiki
wiki. (If
someone could tell me the appropriate page to use
that would
be useful. Otherwise, I will just create a page in
my own
namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of
lines in
those files: 85606 Lines covered: 59489 Lines not covered:
26117
Percentage covered: 0.694916244188
So, parserTests is getting (at best) about 70%
code
coverage. This is better than I expected, but
still it means
parserTests does not test 26117 lines of code.
What I mean
by "at best" is xdebug just notes whether a line
of code is
visited. It doesn't do any logic analysis on which
branches
are taken. Furthermore, parserTests may not visit
some files
that are critical to the operation of the MW
software.
Obviously, xdebug can only gather statistics on
visited
files.
I want to emphasize that there may be errors in
these
results due to bad assumptions on my part or bad
coding.
However, it is a place to start.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
=
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
dan nessett wrote:
I decided to investigate how well parserTests exercises the MW code. So, I threw together a couple of MacGyver tools that use xdebug's code coverage capability and analyzed the results. The results are very, very preliminary, but I thought I would get them out so others can look them over. In the next couple of days I hope to post more detailed results and the tools themselves on the Mediawiki wiki. (If someone could tell me the appropriate page to use that would be useful. Otherwise, I will just create a page in my own namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of lines in those files: 85606 Lines covered: 59489 Lines not covered: 26117 Percentage covered: 0.694916244188
So, parserTests is getting (at best) about 70% code coverage. This is better than I expected, but still it means parserTests does not test 26117 lines of code. What I mean by "at best" is xdebug just notes whether a line of code is visited. It doesn't do any logic analysis on which branches are taken. Furthermore, parserTests may not visit some files that are critical to the operation of the MW software. Obviously, xdebug can only gather statistics on visited files.
I want to emphasize that there may be errors in these results due to bad assumptions on my part or bad coding. However, it is a place to start.
Well, they are *parser* tests, they are not intended to cover Special:Version or something else.
--vvv
True. However, knowing the coverage of parserTests and knowing which code isn't even being visited by it is the first step in understanding where the holes are in testing. Code coverage is a primitive metric. But, it's a place to start.
--- On Thu, 7/30/09, Victor Vasiliev vasilvv@gmail.com wrote:
From: Victor Vasiliev vasilvv@gmail.com Subject: Re: [Wikitech-l] parserTests code coverage statistics To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Thursday, July 30, 2009, 1:28 AM dan nessett wrote:
I decided to investigate how well parserTests
exercises the MW code. So, I threw together a couple of MacGyver tools that use xdebug's code coverage capability and analyzed the results. The results are very, very preliminary, but I thought I would get them out so others can look them over. In the next couple of days I hope to post more detailed results and the tools themselves on the Mediawiki wiki. (If someone could tell me the appropriate page to use that would be useful. Otherwise, I will just create a page in my own namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of lines
in those files: 85606
Lines covered: 59489 Lines not covered:
26117 Percentage covered: 0.694916244188
So, parserTests is getting (at best) about 70% code
coverage. This is better than I expected, but still it means parserTests does not test 26117 lines of code. What I mean by "at best" is xdebug just notes whether a line of code is visited. It doesn't do any logic analysis on which branches are taken. Furthermore, parserTests may not visit some files that are critical to the operation of the MW software. Obviously, xdebug can only gather statistics on visited files.
I want to emphasize that there may be errors in these
results due to bad assumptions on my part or bad coding. However, it is a place to start.
Well, they are *parser* tests, they are not intended to cover Special:Version or something else.
--vvv
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Jul 30, 2009 at 10:53 AM, dan nessettdnessett@yahoo.com wrote:
True. However, knowing the coverage of parserTests and knowing which code isn't even being visited by it is the first step in understanding where the holes are in testing. Code coverage is a primitive metric. But, it's a place to start.
--- On Thu, 7/30/09, Victor Vasiliev vasilvv@gmail.com wrote:
From: Victor Vasiliev vasilvv@gmail.com Subject: Re: [Wikitech-l] parserTests code coverage statistics To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Thursday, July 30, 2009, 1:28 AM dan nessett wrote:
I decided to investigate how well parserTests
exercises the MW code. So, I threw together a couple of MacGyver tools that use xdebug's code coverage capability and analyzed the results. The results are very, very preliminary, but I thought I would get them out so others can look them over. In the next couple of days I hope to post more detailed results and the tools themselves on the Mediawiki wiki. (If someone could tell me the appropriate page to use that would be useful. Otherwise, I will just create a page in my own namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of lines
in those files: 85606
Lines covered: 59489 Lines not covered:
26117 Percentage covered: 0.694916244188
So, parserTests is getting (at best) about 70% code
coverage. This is better than I expected, but still it means parserTests does not test 26117 lines of code. What I mean by "at best" is xdebug just notes whether a line of code is visited. It doesn't do any logic analysis on which branches are taken. Furthermore, parserTests may not visit some files that are critical to the operation of the MW software. Obviously, xdebug can only gather statistics on visited files.
I want to emphasize that there may be errors in these
results due to bad assumptions on my part or bad coding. However, it is a place to start.
Well, they are *parser* tests, they are not intended to cover Special:Version or something else.
--vvv
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
For more generic unit tests, check out the stuff in /t/ and /tests/ Those could probably use improvement.
-Chad
Right. I have looked at both t/ and tests/ and agree that they could use some work. But when starting on a trip its best to walk in one direction to start. Otherwise you end up going around in circles.
--- On Thu, 7/30/09, Chad innocentkiller@gmail.com wrote:
From: Chad innocentkiller@gmail.com Subject: Re: [Wikitech-l] parserTests code coverage statistics To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Thursday, July 30, 2009, 8:18 AM On Thu, Jul 30, 2009 at 10:53 AM, dan nessettdnessett@yahoo.com wrote:
True. However, knowing the coverage of parserTests and
knowing which code isn't even being visited by it is the first step in understanding where the holes are in testing. Code coverage is a primitive metric. But, it's a place to start.
--- On Thu, 7/30/09, Victor Vasiliev vasilvv@gmail.com
wrote:
From: Victor Vasiliev vasilvv@gmail.com Subject: Re: [Wikitech-l] parserTests code
coverage statistics
To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Date: Thursday, July 30, 2009, 1:28 AM dan nessett wrote:
I decided to investigate how well
parserTests
exercises the MW code. So, I threw together a
couple of
MacGyver tools that use xdebug's code coverage
capability
and analyzed the results. The results are very,
very
preliminary, but I thought I would get them out so
others
can look them over. In the next couple of days I
hope to
post more detailed results and the tools
themselves on the
Mediawiki wiki. (If someone could tell me the
appropriate
page to use that would be useful. Otherwise, I
will just
create a page in my own namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of
lines
in those files: 85606
Lines covered: 59489 Lines not covered:
26117 Percentage covered: 0.694916244188
So, parserTests is getting (at best) about
70% code
coverage. This is better than I expected, but
still it means
parserTests does not test 26117 lines of code.
What I mean
by "at best" is xdebug just notes whether a line
of code is
visited. It doesn't do any logic analysis on which
branches
are taken. Furthermore, parserTests may not visit
some files
that are critical to the operation of the MW
software.
Obviously, xdebug can only gather statistics on
visited
files.
I want to emphasize that there may be errors
in these
results due to bad assumptions on my part or bad
coding.
However, it is a place to start.
Well, they are *parser* tests, they are not
intended to
cover Special:Version or something else.
--vvv
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
For more generic unit tests, check out the stuff in /t/ and /tests/ Those could probably use improvement.
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 30/07/2009, at 3:53 PM, dan nessett wrote:
True. However, knowing the coverage of parserTests and knowing which code isn't even being visited by it is the first step in understanding where the holes are in testing. Code coverage is a primitive metric. But, it's a place to start.
The parser test is a test of the parser. The parser tests do not, have not, and will not cover anything other than the parser.
Measuring code coverage as a percentage of *core* is a really bad idea. Try checking the percentage of includes/parser/ covered.
-- Andrew Garrett agarrett@wikimedia.org http://werdn.us/
wikitech-l@lists.wikimedia.org