Hi,
here are my thoughts about phpunit and selenium testing.
The wiki under test is set up with a master database
consisting of a single objectcache table.
The entries of this table specify a test run identifier as primary key
and temporary resource identifiers as dependent fields.
If I understand this
correctly, this would not allow to test any wikis that are running on live sites, e.g.
intranet wikis. While I agree that regression testing on live sites is not a good idea, I
kind of like the notion that after setting up a wiki with all the extensions I like to
have, I could do some sort of "everything up and running"-test. With the concept
of using separate testing databases and resources, this would be possible without
interference with the actual data and could even be done at intervals during, say,
maintenance periods.
Setup of a test run requires the creation of the test
run temporary resources and a entry in the objectcache table.
Are there already
mechanisms for this? I haven't done too much work with the objectcache. This is where
memcached data is stored, right? So how do I get the data that is needed? This question
leads me to another one: How do I get the testind database and resources? As I see this,
it should be part of the testing framework to be able to produce the set of data needed
from a "normal" MW installation. The whole mechanism would actually be something
like a backup, so we might look into any existing solutions for that.
When a request is sent to the wiki under test, very
early in the request
processing (e.g., immediately after LocalSettings is processed) a hook is
called with the provided state information as an argument that accesses
the objectcache table. The extension function handling the hook switches
in the temporary resources and returns.
Also, this hook might handle the
reconfiguration of the wiki, if needed. So for example, testing the PagedTiffHandler
requires uploading of tiff files to be enabled. However, there might be some security
risks, since it is not directly obvious in the code which settings are changed. So the
hook should only be called when $wgEnableSelenium = true. In addition, we could define a
DontTouchThisSettings.php, which is called even after the hook and holds some settings
that are holy to the admin of the wiki :) The question is, though, would this not become
somewhat too complicated?
After the test run completes, the testing application
cleans up the test run
by requesting the deletion of the temporary resources and the objectcache table
entry associated with the test run.
In some cases, tests will not change any data,
e.g. testing dynamic skin elements in vector skin. Would it make sense not to tear down
the testing environment in that case in order to save some time when testing repeatedly? I
think, there is a conflict between performance and amount of data, but who wins?
In general, it seems to me that we have some similarity with what is called wiki family on
mediawiki.org. One could see multiple testing environments as a set of multiple wikis that
share a common codebase [1]. Does anybody have experience with wiki families and the
object cache rsp. memcached?
I am not sure whether we can use the same codebase as parsertests. I'd rather think,
parser tests are a special case of what we are sketching here. On the other hand, I
don't think it is a good idea to have two separate approaches for very similar tasks
in the code. Do you think it would be feasible to separate the preparation part from both
parser tests and selenium tests and build both of them on a common ground?
Best regards,
Markus
[1]
http://www.mediawiki.org/wiki/Wiki_family#Scenario_2:_Quick_set-up