dan nessett dnessett@yahoo.com wrote:
I am investigating how to write a comprehensive parser regression test. What I mean by this is something you wouldn't normally run frequently, but rather something that we could use to get past the "known to fail" tests now disabled. The problem is no one understands the parser well enough to have confidence that if you fix one of these tests that you will not break something else.
So, I thought, how about using the guts of DumpHTML to create a comprehensive parser regression test. The idea is to have two versions of phase3 + extensions, one without the change you make to the parser to fix a known-to-fail test (call this Base) and one with the change (call this Current). Modify DumpHTML to first visit a page through Base, saving the HTML then visit the same page through Current and compare the two results. Do this for every page in the database. If there are no differences, the change in Current works. [...]
I use a similar approach on a toolserver script in addition to smaller tests: I saved several revisions of "interesting" wiki pages and the respective output of the then-current script version to the subversion repository. Before commit- ting changes, I run a test whether the current script produ- ces the same results. If the results are different, either a bug needs to be fixed or the expected output be amended (in the same commit).
Tim
P. S.: I find your dedication to QA very laudable; I think though that more people would read and embrace your thoughts if you would find a more concise way to put them across :-).