Hmm, I think you guys are overthinking the details on
this; let's step
back a level.
When you're running tests, you have these tasks: * create a blank-slate
wiki to run tests in * populate the empty wiki with known data * run
tests on this wiki in a known state * clean up in some way
The parserTests system is designed with some particular constraints: *
run within the local MediaWiki test instance a developer already has,
without altering its contents
* run all tests in a single command-line program * expose no in-test
data to the outside world
It can do this very easily using temporary tables and a temporary
directory because it only needs to work within that one single test
process; once it's done, it's done and all the temporary data can be
discarded. Nothing needs to be kept across processes or exposed to
clients.
For Selenium tests, you have a very different set of constraints: * test
data must be exposed to web clients over a web server * test data must
be retained across multiple requests
The simplest way to accomplish this is to have a dedicated, web-exposed
test wiki instance. A test run would go like this: * (re)initialize test
wiki into known state * run series of tests
I'd recommend not trying to use the parserTest harness/initialization
for this; it'll be a *lot* simpler to just script creation of a fresh
wiki. (drop database, create database, slurp tables, run update.php)
Some non-destructive tests can always be run on any existing instance --
and probably should be! -- and some 'active' tests will be freely
runnable on existing instances that are used for development and
testing, but if you want to work with a blank slate wiki exposed to web
clients, keep things simple and just make a dedicated instance.
-- brion
On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett <dnessett(a)yahoo.com> wrote:
> On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote:
>
> > Hi,
> >
> > here are my thoughts about phpunit and selenium testing.
> >
> >> The wiki under test is set up with a master database consisting of a
> >> single objectcache table. The entries of this table specify a test
> >> run identifier as primary key and temporary resource identifiers as
> >> dependent fields.
> > If I understand this correctly, this would not allow to test any
> > wikis that are running on live sites, e.g. intranet wikis. While I
> > agree that regression testing on live sites is not a good idea, I
> > kind of like the notion that after setting up a wiki with all the
> > extensions I like to have, I could do some sort of "everything up and
> > running"-test. With the concept of using separate testing databases
> > and resources, this would be possible without interference with the
> > actual data and could even be done at intervals during, say,
> > maintenance periods.
>
> The problem with testing live sites is tests may alter wiki data
> (consequently, test run reproducibility becomes a problem). If all of
> the tests are read-only, then that isn't a problem, but it means
> developing a whole set of tests that conform to that constraint.
>
> Nevertheless, it wouldn't be hard to design the switching mechanism to
> allow the testing of live sites. There could be an option in test setup
> and cleanup that effectively says "don't switch-in/clean-up temporary
> resources." Personally, I think use of this option is dangerous, but it
> wouldn't be hard to implement.
>
> >> Setup of a test run requires the creation of the test run temporary
> >> resources and a entry in the objectcache table.
> > Are there already mechanisms for this? I haven't done too much work
> > with the objectcache. This is where memcached data is stored, right?
> > So how do I get the data that is needed? This question leads me to
> > another one: How do I get the testind database and resources? As I
> > see this, it should be part of the testing framework to be able to
> > produce the set of data needed from a "normal" MW installation. The
> > whole mechanism would actually be something like a backup, so we
> > might look into any existing solutions for that.
>
> We should use the existing ObjectCache class to manage the object
> cache. However, if there exists some switching-in code, I doubt it has
> corresponding clean-up code. So, we probably need to do some
> development even if we use existing mechanisms.
>
> I think the object cache and memcached are alternative ways of storing
> persistent data. (I also am not an expert in this, so I could be
> wrong). My understanding is memcached uses the memcached daemon
> (http://
memcached.org/), while the object cache uses the underlying
> database. If so, then memcached data disappears after a system crash or
> power outage, whereas object cache data should survive.
>
> You are absolutely correct that we need to figure out how to clone a
> set of temporary resources (db, images directory, perhaps cache data)
> and set them up for use (i.e., so the switch-in logic can copy them for
> the test run). There are a number of problems to solve, e.g., 1) how do
> you package the cloned resources (e.g., tar file), 2) how do you
> efficiently copy them during the switch-in process, 3) security issues,
> 4) resource management issues.
>
> >> When a request is sent to the wiki under test, very early in the
> >> request processing (e.g., immediately after LocalSettings is
> >> processed) a hook is called with the provided state information as
> >> an argument that accesses the objectcache table. The extension
> >> function handling the hook switches in the temporary resources and
> >> returns.
> > Also, this hook might handle the reconfiguration of the wiki, if
> > needed. So for example, testing the PagedTiffHandler requires
> > uploading of tiff files to be enabled. However, there might be some
> > security risks, since it is not directly obvious in the code which
> > settings are changed. So the hook should only be called when
> > $wgEnableSelenium = true. In addition, we could define a
> > DontTouchThisSettings.php, which is called even after the hook and
> > holds some settings that are holy to the admin of the wiki :) The
> > question is, though, would this not become somewhat too complicated?
>
> I have done some research on possible hooks. I think the
> SetupAfterCache hook is a good candidate. The problem with calling a
> (psuedo-)hook in LocalSettings is there is some setup not completed
> when LocatSettings is processed (e.g., the WebRequest object is not yet
> available. the ObjectCache class file is not yet included). Most setup
> is completed before calling the SetupAfterCache hook.
>
> It would be possible to do some configuration in this hook, but I think
> we need to consider efficiency. Preferably, the creation of the base
> resources that are copied during the switch-in will be configured as
> much as possible. For example, I think creating the base for
> PagedTiffHandler should start with a freshly installed wiki and upload
> multipage.tiff. The resulting db could be dumped and the images
> directory as well as other resources copied. The result could them be
> tar'd and uploaded as the base for the PagedTiffHandler test suite.
> This would relieve the test set up code of uploading multipage.tiff on
> each test run.
>
> We may even consider pre-installing the base db so that the switch-in
> code can use db functions to clone it, rather than creating it from the
> base dump for each test run that uses it. The latter could take a
> significant amount of time, thereby severely slowing testing.
>
> >> After the test run completes, the testing application cleans up the
> >> test run by requesting the deletion of the temporary resources and
> >> the objectcache table entry associated with the test run.
> > In some cases, tests will not change any data, e.g. testing dynamic
> > skin elements in vector skin. Would it make sense not to tear down
> > the testing environment in that case in order to save some time when
> > testing repeatedly? I think, there is a conflict between performance
> > and amount of data, but who wins?
>
> There may be ways to make things more efficient by not immediately
> deleting the temporary resources. However, eventually we have to delete
> them. So, there is a question of where the temporary resources
> identifier is stored so it can be used later for a clean-up request. I
> was assuming the switch-in request occurs in test suite start-up and
> the clean-up request occurs in the test suite finishing code. But, we
> should consider alternative implementation strategies.
>
> > In general, it seems to me that we have some similarity with what is
> > called wiki family on
mediawiki.org. One could see multiple testing
> > environments as a set of multiple wikis that share a common codebase
> > [1]. Does anybody have experience with wiki families and the object
> > cache rsp. memcached?
>
> I agree. (In fact, I mentioned this previously). No need to reinvent
> the wheel.
>
> > I am not sure whether we can use the same codebase as parsertests.
> > I'd rather think, parser tests are a special case of what we are
> > sketching here. On the other hand, I don't think it is a good idea to
> > have two separate approaches for very similar tasks in the code. Do
> > you think it would be feasible to separate the preparation part from
> > both parser tests and selenium tests and build both of them on a
> > common ground?
> >
> >
> The parserTests code creates some temporary tables in an existing wiki
> database. Currently, these tables are empty and prefixed with a static
> identifier. The requirements for parserTests and selenium tests are
> significantly different. While we may be able to learn from the
> parserTests code, we would have to change the parserTest code
> significantly in order to use it. I think it would actually be less
> work to start from scratch. Of course, as you mention, there may be
> code used to support wiki families that we could use without much
> modification.
>
> --
> -- Dan Nessett
>
>
> _______________________________________________ Wikitech-l mailing list
> Wikitech-l(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
100% agreement.
Some have mentioned the possibility of using the wiki family logic to
help achieve these objectives. Do you have any thoughts on this? If you
think it is a good idea, how do we find out more about it?
--
-- Dan Nessett