MW QA - Wikitech-l

16 Jul 2009

I have never been a QA engineer. However, it doesn't require great experience to see
that the MW software development process is broken. I provide the following comments not
in a destructive spirit. The success of the MW software is obvious. However, in my view
unless the development process introduces some QA procedures, the code eventually will
collapse and its reputation will degrade.

My interest in MW (the software, not the organization) is driven by a desire to provide an
enhancement in the form of an extension. So, I began by building a small development
environment on my machine (a work in progress). Having developed software for other
organizations, I intuitively sought out what I needed in terms of testing in order to
provide a good quality extension. This meant I needed to develop unit tests for my
extension and also to perform regression testing on the main code base after installing
it. Hence some of my previous questions to this email list.

It soon became apparent that the MW development process has little or no testing
procedures. Sure, there are the parser tests, but I couldn't find any requirement that
developers had to run them before submitting patches.

Out of curiosity, I decided to download 1.16a (r52088), use the LocalSettings file from my
local installation (1.14) and run some parser tests. This is not a scientific experiment,
since the only justification for using these extensions in the tests is I had them
installed in my personal wiki. However, there is at least one thing to learn from them.
The results are:

Mediawiki 52088 Parser Tests

Extensions : 1) Nuke, 2) Renameuser, 3) Cite, 4) ParserFunctions, 5) CSS Style Sheets, 6)
ExpandTemplates, 7) Gadgets, 8) Dynamic Page List, 9) Labeled Section Transclusion. The
last extension has 3 require_once files: a) lst.php, b) lsth.php, and c) compat.php.

Test	Extensions			ParserTests Test Fails

1	1,2,3,4,5,6,7,8,9	        19
2	1				14
3	2				14
4	3				14
5	4				14
6	5				14
7	6				14
8	7				14
9	8				14
10	9 (abc)				19
11	9 (a)				18
12	9 (ab)				19
13	1,2,3,4,6,7			14

Note that the extension that introduces all of the unexpected parser test failures is
Labeled Section Transclusion. According to its documentation, it is installed on
*.wikisource.org, test.wikipedia.org, and en.wiktionary.org.

I am new to this development community, but my guess is since there are no testing
requirements for extensions, its author did not run parser tests before publishing it. (I
don't mean to slander him and I am open to the correction that it ran without
unexpected errors on the MW version he tested against.)

This rather long preamble leads me to the point of this email. The MW software development
process needs at least some rudimentary QA procedures. Here are some thoughts on this. I
offer these to initiate debate on this issue, not as hard positions.

* Before a developer commits a patch to the code base, he must run parser tests against
the change. The patch should not be committed if it increases the number of parser test
failures. He should document the results in the bugzilla bug report.

* If a developer commits a patch without running parser tests or commits a patch that
increases the number of parser test failures, he should be warned. If he does this another
time with some time interval (say 6 months), his commit privileges are revoked for some
period of time (say 6 months). The second time he becomes a candidate for commit privilege
revocation, they will be revoked permanently.

* An extension developer also should run parser tests against a MW version with the
extension installed. The results of this should be provided in the extension
documentation. An extension should not be added to the extension matrix unless it provides
this information.

* A test harness that performs regression tests (currently only parser tests) against
every trunk versions committed in the last 24 hours should be run nightly. The installed
extensions should be those used on the WMF machines. The results should be published on
some page on the Mediawiki site. If any version increases the number of parser test
failures, the procedure described above for developers is initiated.

* A group of developers should have the responsibility of reviewing the nightly test
results to implement this QA process.

I am sure there are a whole bunch of other things that might be done to improve MW QA. The
point of this message is to initiate a discussion on what those might be.