True. However, knowing the coverage of parserTests and knowing which code isn't even
being visited by it is the first step in understanding where the holes are in testing.
Code coverage is a primitive metric. But, it's a place to start.
--- On Thu, 7/30/09, Victor Vasiliev <vasilvv(a)gmail.com> wrote:
From: Victor Vasiliev <vasilvv(a)gmail.com>
Subject: Re: [Wikitech-l] parserTests code coverage statistics
To: "Wikimedia developers" <wikitech-l(a)lists.wikimedia.org>
Date: Thursday, July 30, 2009, 1:28 AM
dan nessett wrote:
I decided to investigate how well parserTests
exercises the MW code. So, I threw together a couple of
MacGyver tools that use xdebug's code coverage capability
and analyzed the results. The results are very, very
preliminary, but I thought I would get them out so others
can look them over. In the next couple of days I hope to
post more detailed results and the tools themselves on the
Mediawiki wiki. (If someone could tell me the appropriate
page to use that would be useful. Otherwise, I will just
create a page in my own namespace).
The statistics (again very preliminary) are:
Number of files exercised: 141 Number of lines
in those files: 85606
Lines covered: 59489 Lines not covered:
26117 Percentage covered: 0.694916244188
So, parserTests is getting (at best) about 70% code
coverage. This is better than
I expected, but still it means
parserTests does not test 26117 lines of code. What I mean
by "at best" is xdebug just notes whether a line of code is
visited. It doesn't do any logic analysis on which branches
are taken. Furthermore, parserTests may not visit some files
that are critical to the operation of the MW software.
Obviously, xdebug can only gather statistics on visited
files.
I want to emphasize that there may be errors in these
results due to bad
assumptions on my part or bad coding.
However, it is a place to start.
Well, they are *parser* tests, they are not intended to
cover
Special:Version or something else.
--vvv
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l