dan nessett <dnessett(a)yahoo.com> wrote:
I am investigating how to write a comprehensive parser
regression test. What I mean by this is something you wouldn't normally run
frequently, but rather something that we could use to get past the "known to
fail" tests now disabled. The problem is no one understands the parser well enough to
have confidence that if you fix one of these tests that you will not break something else.
So, I thought, how about using the guts of DumpHTML to
create a comprehensive parser regression test. The idea is to have two versions of phase3
+ extensions, one without the change you make to the parser to fix a known-to-fail test
(call this Base) and one with the change (call this Current). Modify DumpHTML to first
visit a page through Base, saving the HTML then visit the same page through Current and
compare the two results. Do this for every page in the database. If there are no
differences, the change in Current works.
[...]
I use a similar approach on a toolserver script in addition
to smaller tests: I saved several revisions of "interesting"
wiki pages and the respective output of the then-current
script version to the subversion repository. Before commit-
ting changes, I run a test whether the current script produ-
ces the same results. If the results are different, either a
bug needs to be fixed or the expected output be amended (in
the same commit).
Tim
P. S.: I find your dedication to QA very laudable; I think
though that more people would read and embrace your
thoughts if you would find a more concise way to put
them across :-).