Hmm, I think you guys are overthinking the details on this; let's step back
When you're running tests, you have these tasks:
* create a blank-slate wiki to run tests in
* populate the empty wiki with known data
* run tests on this wiki in a known state
* clean up in some way
The parserTests system is designed with some particular constraints:
* run within the local MediaWiki test instance a developer already has,
without altering its contents
* run all tests in a single command-line program
* expose no in-test data to the outside world
It can do this very easily using temporary tables and a temporary directory
because it only needs to work within that one single test process; once it's
done, it's done and all the temporary data can be discarded. Nothing needs
to be kept across processes or exposed to clients.
For Selenium tests, you have a very different set of constraints:
* test data must be exposed to web clients over a web server
* test data must be retained across multiple requests
The simplest way to accomplish this is to have a dedicated, web-exposed test
wiki instance. A test run would go like this:
* (re)initialize test wiki into known state
* run series of tests
I'd recommend not trying to use the parserTest harness/initialization for
this; it'll be a *lot* simpler to just script creation of a fresh wiki.
(drop database, create database, slurp tables, run update.php)
Some non-destructive tests can always be run on any existing instance -- and
probably should be! -- and some 'active' tests will be freely runnable on
existing instances that are used for development and testing, but if you
want to work with a blank slate wiki exposed to web clients, keep things
simple and just make a dedicated instance.
On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett <dnessett(a)yahoo.com> wrote:
On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser
here are my thoughts about phpunit and selenium testing.
The wiki under test is set up with a master
database consisting of a
single objectcache table. The entries of this table specify a test run
identifier as primary key and temporary resource identifiers as
If I understand this correctly, this would not allow to test any
that are running on live sites, e.g. intranet wikis. While I agree that
regression testing on live sites is not a good idea, I kind of like the
notion that after setting up a wiki with all the extensions I like to
have, I could do some sort of "everything up and running"-test. With the
concept of using separate testing databases and resources, this would be
possible without interference with the actual data and could even be
done at intervals during, say, maintenance periods.
The problem with testing live sites is tests may alter wiki data
(consequently, test run reproducibility becomes a problem). If all of the
tests are read-only, then that isn't a problem, but it means developing a
whole set of tests that conform to that constraint.
Nevertheless, it wouldn't be hard to design the switching mechanism to
allow the testing of live sites. There could be an option in test setup
and cleanup that effectively says "don't switch-in/clean-up temporary
resources." Personally, I think use of this option is dangerous, but it
wouldn't be hard to implement.
a test run requires the creation of the test run temporary
resources and a entry in the objectcache table.
Are there already mechanisms for
this? I haven't done too much work with
the objectcache. This is where memcached data is stored, right? So how
do I get the data that is needed? This question leads me to another one:
How do I get the testind database and resources? As I see this, it
should be part of the testing framework to be able to produce the set of
data needed from a "normal" MW installation. The whole mechanism would
actually be something like a backup, so we might look into any existing
solutions for that.
We should use the existing ObjectCache class to manage the object cache.
However, if there exists some switching-in code, I doubt it has
corresponding clean-up code. So, we probably need to do some development
even if we use existing mechanisms.
I think the object cache and memcached are alternative ways of storing
persistent data. (I also am not an expert in this, so I could be wrong).
My understanding is memcached uses the memcached daemon (http://
), while the object cache uses the underlying database. If
so, then memcached data disappears after a system crash or power outage,
whereas object cache data should survive.
You are absolutely correct that we need to figure out how to clone a set
of temporary resources (db, images directory, perhaps cache data) and set
them up for use (i.e., so the switch-in logic can copy them for the test
run). There are a number of problems to solve, e.g., 1) how do you
package the cloned resources (e.g., tar file), 2) how do you efficiently
copy them during the switch-in process, 3) security issues, 4) resource
request is sent to the wiki under test, very early in the
request processing (e.g., immediately after LocalSettings is processed)
a hook is called with the provided state information as an argument
that accesses the objectcache table. The extension function handling
the hook switches in the temporary resources and returns.
Also, this hook might
handle the reconfiguration of the wiki, if needed.
So for example, testing the PagedTiffHandler requires uploading of tiff
files to be enabled. However, there might be some security risks, since
it is not directly obvious in the code which settings are changed. So
the hook should only be called when $wgEnableSelenium = true. In
addition, we could define a DontTouchThisSettings.php, which is called
even after the hook and holds some settings that are holy to the admin
of the wiki :) The question is, though, would this not become somewhat
I have done some research on possible hooks. I think the SetupAfterCache
hook is a good candidate. The problem with calling a (psuedo-)hook in
LocalSettings is there is some setup not completed when LocatSettings is
processed (e.g., the WebRequest object is not yet available. the
ObjectCache class file is not yet included). Most setup is completed
before calling the SetupAfterCache hook.
It would be possible to do some configuration in this hook, but I think
we need to consider efficiency. Preferably, the creation of the base
resources that are copied during the switch-in will be configured as much
as possible. For example, I think creating the base for PagedTiffHandler
should start with a freshly installed wiki and upload multipage.tiff. The
resulting db could be dumped and the images directory as well as other
resources copied. The result could them be tar'd and uploaded as the base
for the PagedTiffHandler test suite. This would relieve the test set up
code of uploading multipage.tiff on each test run.
We may even consider pre-installing the base db so that the switch-in
code can use db functions to clone it, rather than creating it from the
base dump for each test run that uses it. The latter could take a
significant amount of time, thereby severely slowing testing.
test run completes, the testing application cleans up the
test run by requesting the deletion of the temporary resources and the
objectcache table entry associated with the test run.
In some cases, tests will
not change any data, e.g. testing dynamic skin
elements in vector skin. Would it make sense not to tear down the
testing environment in that case in order to save some time when testing
repeatedly? I think, there is a conflict between performance and amount
of data, but who wins?
There may be ways to make things more efficient by not immediately
deleting the temporary resources. However, eventually we have to delete
them. So, there is a question of where the temporary resources identifier
is stored so it can be used later for a clean-up request. I was assuming
the switch-in request occurs in test suite start-up and the clean-up
request occurs in the test suite finishing code. But, we should consider
alternative implementation strategies.
In general, it seems to me that we have some
similarity with what is
called wiki family on mediawiki.org
. One could see multiple testing
environments as a set of multiple wikis that share a common codebase
. Does anybody have experience with wiki families and the object
cache rsp. memcached?
I agree. (In fact, I mentioned this previously). No need to reinvent the
I am not sure whether we can use the same
codebase as parsertests. I'd
rather think, parser tests are a special case of what we are sketching
here. On the other hand, I don't think it is a good idea to have two
separate approaches for very similar tasks in the code. Do you think it
would be feasible to separate the preparation part from both parser
tests and selenium tests and build both of them on a common ground?
The parserTests code creates some temporary tables in an existing wiki
database. Currently, these tables are empty and prefixed with a static
identifier. The requirements for parserTests and selenium tests are
significantly different. While we may be able to learn from the
parserTests code, we would have to change the parserTest code
significantly in order to use it. I think it would actually be less work
to start from scratch. Of course, as you mention, there may be code used
to support wiki families that we could use without much modification.
-- Dan Nessett
Wikitech-l mailing list