I have been tasked to evaluate whether we can use the parserTests db code for the selenium framework. I just looked it over and have serious reservations. I would appreciate any comments on the following analysis. The environment for selenium tests is different than that for parserTests. It is envisioned that multiple concurrent tests could run using the same MW code base. Consequently, each test run must: + Use a db that if written to will not destroy other test wiki information. + Switch in a new images and math directory so any writes do not interfere with other tests. + Maintain the integrity of the cache. Note that tests would *never* run on a production wiki (it may be possible to do so if they do no writes, but safety considerations suggest they should always run on a test data, not production data). In fact production wikis should always retain the setting $wgEnableSelenium = false, to ensure selenium test are disabled. Given this background, consider the following (and feel free to comment on it): parserTests temporary table code: A fixed set of tables are specified in the code. parserTests creates temporary tables with the same name, but using a different static prefix. These tables are used for the parserTests run. Problems using this approach for selenium tests: + Selenium tests on extensions may require use of extension specific tables, the names of which cannot be elaborated in the code. + Concurrent test runs of parserTests are not supported, since the temporary tables have fixed names and therefore concurrent writes to them by parallel test runs would cause interference. + Clean up from aborted runs requires dropping fossil tables. But, if a previous run tested an extension with extension-specific tables, there is no way for a test of some other functionality to figure out which tables to drop. For these reasons, I don't think we can reuse the parserTests code. However, I am open to arguments to the contrary.

After reflection, here are some other problems. + Some tests assume the existence of data in the db. For example, the PagedTiffHandler tests assume the image Multipage.tiff is already loaded. However, this requires an entry in the image table. You could modify the test to clone the existing image table, but that means you have problems with: + Some tests assume certain data is *not* in the db. PagedTiffHandler has tests that upload images. These cannot already be in the images table. So, you can't simply clone the images table.

All of this suggests to me that a better strategy is: + When the test run begins, clone a db associated with the test suite.

+ Switch the wiki to use this db and return a cookie or some other state information that identifies this test run configuration.

On Sat, 18 Sep 2010 00:53:04 +0200, Platonides wrote: Not sure what you mean by remote petitions.

Selenium requests always come through the web portal from the selenium server. So, no internal queries are involved.

Where to store the data is an open question, one that requires consultation with others. However, here are some thoughts: + The data must be persistent. If the wiki crashes for some reason, there may be cloned dbs and test-specific copies of images and images/math hanging around. (Depending how we handle the cache information, there may also be fossil cache data). This requires cleanup after a wiki crash. + It would be possible to store the data in a file or in a master db table. Which is best (or if something else is better) is a subject for discussion.

The storage has to be persistent to accommodate wiki crashes (e.g., httpd crash, server OS crash, power outage). It might be possible to use memcachedb, but as far as I am aware that requires installing Berkeley DB, which complicated deployment. Why not employ the already installed DB software used by the wiki? That provides persistent storage and requires no additional software.

Dan Nessett wrote: My original idea was to use whatever ObjectCache the wiki used, but it could be forced to use the db as backend (that's the objectcache table).

Dan Nessett wrote: You load originaldb.objectcache, retrieve the specific configuration, and switch into it. For supporting many sumyltaneous configurations, the keyname could have the instance (whatever that cookie is set to) appended, although those dynamic configurations make me a bit nervous.

Dan Nessett wrote: Such originaldb would be empty for an instance used just for regression testing and could in fact only contain the objectcache table. If it's a developer machine he would use the originaldb for local testing, but a nigthly would not need to (in fact, errors trying to access those missing tables would be useful for detecting errors in the isolating system).

Hi, here are my thoughts about phpunit and selenium testing. If I understand this correctly, this would not allow to test any wikis that are running on live sites, e.g. intranet wikis. While I agree that regression testing on live sites is not a good idea, I kind of like the notion that after setting up a wiki with all the extensions I like to have, I could do some sort of "everything up and running"-test. With the concept of using separate testing databases and resources, this would be possible without interference with the actual data and could even be done at intervals during, say, maintenance periods.

Are there already mechanisms for this? I haven't done too much work with the objectcache. This is where memcached data is stored, right? So how do I get the data that is needed? This question leads me to another one: How do I get the testind database and resources? As I see this, it should be part of the testing framework to be able to produce the set of data needed from a "normal" MW installation. The whole mechanism would actually be something like a backup, so we might look into any existing solutions for that.

Also, this hook might handle the reconfiguration of the wiki, if needed. So for example, testing the PagedTiffHandler requires uploading of tiff files to be enabled. However, there might be some security risks, since it is not directly obvious in the code which settings are changed. So the hook should only be called when $wgEnableSelenium = true. In addition, we could define a DontTouchThisSettings.php, which is called even after the hook and holds some settings that are holy to the admin of the wiki :) The question is, though, would this not become somewhat too complicated?

In some cases, tests will not change any data, e.g. testing dynamic skin elements in vector skin. Would it make sense not to tear down the testing environment in that case in order to save some time when testing repeatedly? I think, there is a conflict between performance and amount of data, but who wins?

In general, it seems to me that we have some similarity with what is called wiki family on mediawiki.org. One could see multiple testing environments as a set of multiple wikis that share a common codebase [1]. Does anybody have experience with wiki families and the object cache rsp. memcached?

I am not sure whether we can use the same codebase as parsertests. I'd rather think, parser tests are a special case of what we are sketching here. On the other hand, I don't think it is a good idea to have two separate approaches for very similar tasks in the code. Do you think it would be feasible to separate the preparation part from both parser tests and selenium tests and build both of them on a common ground?

On Wed, Sep 22, 2010 at 6:47 PM, Dan Nessett <dnessett(a)yahoo.com> wrote: There is no ObjectCache class. Basically, there is a common base class called BagOStuff and from that various classes for various backends are defined such as SqlBagOStuff and MemcacheBagOStuff. To the code outside that class, there is no visible difference. Bryan

2010/9/22 Dan Nessett <dnessett(a)yahoo.com>om>: Either memcached.php, MemCached.php or MWMemcached.php, I forget. The class name is MWMemcached. Roan Kattouw (Catrope)

I just looked at SqlBagOStuff. It already has entry expiration logic. So, it seems perfect for use by the switch-in/clean-up functionality.

Hmm, I think you guys are overthinking the details on this; let's step back a level. When you're running tests, you have these tasks: * create a blank-slate wiki to run tests in * populate the empty wiki with known data * run tests on this wiki in a known state * clean up in some way The parserTests system is designed with some particular constraints: * run within the local MediaWiki test instance a developer already has, without altering its contents * run all tests in a single command-line program * expose no in-test data to the outside world It can do this very easily using temporary tables and a temporary directory because it only needs to work within that one single test process; once it's done, it's done and all the temporary data can be discarded. Nothing needs to be kept across processes or exposed to clients. For Selenium tests, you have a very different set of constraints: * test data must be exposed to web clients over a web server * test data must be retained across multiple requests The simplest way to accomplish this is to have a dedicated, web-exposed test wiki instance. A test run would go like this: * (re)initialize test wiki into known state * run series of tests I'd recommend not trying to use the parserTest harness/initialization for this; it'll be a *lot* simpler to just script creation of a fresh wiki. (drop database, create database, slurp tables, run update.php) Some non-destructive tests can always be run on any existing instance -- and probably should be! -- and some 'active' tests will be freely runnable on existing instances that are used for development and testing, but if you want to work with a blank slate wiki exposed to web clients, keep things simple and just make a dedicated instance. -- brion On Wed, Sep 22, 2010 at 9:47 AM, Dan Nessett <dnessett(a)yahoo.com> wrote: > On Wed, 22 Sep 2010 15:49:40 +0200, Markus Glaser wrote: > > > Hi, > > > > here are my thoughts about phpunit and selenium testing. > > > >> The wiki under test is set up with a master database consisting of a > >> single objectcache table. The entries of this table specify a test > >> run identifier as primary key and temporary resource identifiers as > >> dependent fields. > > If I understand this correctly, this would not allow to test any > > wikis that are running on live sites, e.g. intranet wikis. While I > > agree that regression testing on live sites is not a good idea, I > > kind of like the notion that after setting up a wiki with all the > > extensions I like to have, I could do some sort of "everything up and > > running"-test. With the concept of using separate testing databases > > and resources, this would be possible without interference with the > > actual data and could even be done at intervals during, say, > > maintenance periods. > > The problem with testing live sites is tests may alter wiki data > (consequently, test run reproducibility becomes a problem). If all of > the tests are read-only, then that isn't a problem, but it means > developing a whole set of tests that conform to that constraint. > > Nevertheless, it wouldn't be hard to design the switching mechanism to > allow the testing of live sites. There could be an option in test setup > and cleanup that effectively says "don't switch-in/clean-up temporary > resources." Personally, I think use of this option is dangerous, but it > wouldn't be hard to implement. > > >> Setup of a test run requires the creation of the test run temporary > >> resources and a entry in the objectcache table. > > Are there already mechanisms for this? I haven't done too much work > > with the objectcache. This is where memcached data is stored, right? > > So how do I get the data that is needed? This question leads me to > > another one: How do I get the testind database and resources? As I > > see this, it should be part of the testing framework to be able to > > produce the set of data needed from a "normal" MW installation. The > > whole mechanism would actually be something like a backup, so we > > might look into any existing solutions for that. > > We should use the existing ObjectCache class to manage the object > cache. However, if there exists some switching-in code, I doubt it has > corresponding clean-up code. So, we probably need to do some > development even if we use existing mechanisms. > > I think the object cache and memcached are alternative ways of storing > persistent data. (I also am not an expert in this, so I could be > wrong). My understanding is memcached uses the memcached daemon > (http:// memcached.org/), while the object cache uses the underlying > database. If so, then memcached data disappears after a system crash or > power outage, whereas object cache data should survive. > > You are absolutely correct that we need to figure out how to clone a > set of temporary resources (db, images directory, perhaps cache data) > and set them up for use (i.e., so the switch-in logic can copy them for > the test run). There are a number of problems to solve, e.g., 1) how do > you package the cloned resources (e.g., tar file), 2) how do you > efficiently copy them during the switch-in process, 3) security issues, > 4) resource management issues. > > >> When a request is sent to the wiki under test, very early in the > >> request processing (e.g., immediately after LocalSettings is > >> processed) a hook is called with the provided state information as > >> an argument that accesses the objectcache table. The extension > >> function handling the hook switches in the temporary resources and > >> returns. > > Also, this hook might handle the reconfiguration of the wiki, if > > needed. So for example, testing the PagedTiffHandler requires > > uploading of tiff files to be enabled. However, there might be some > > security risks, since it is not directly obvious in the code which > > settings are changed. So the hook should only be called when > > $wgEnableSelenium = true. In addition, we could define a > > DontTouchThisSettings.php, which is called even after the hook and > > holds some settings that are holy to the admin of the wiki :) The > > question is, though, would this not become somewhat too complicated? > > I have done some research on possible hooks. I think the > SetupAfterCache hook is a good candidate. The problem with calling a > (psuedo-)hook in LocalSettings is there is some setup not completed > when LocatSettings is processed (e.g., the WebRequest object is not yet > available. the ObjectCache class file is not yet included). Most setup > is completed before calling the SetupAfterCache hook. > > It would be possible to do some configuration in this hook, but I think > we need to consider efficiency. Preferably, the creation of the base > resources that are copied during the switch-in will be configured as > much as possible. For example, I think creating the base for > PagedTiffHandler should start with a freshly installed wiki and upload > multipage.tiff. The resulting db could be dumped and the images > directory as well as other resources copied. The result could them be > tar'd and uploaded as the base for the PagedTiffHandler test suite. > This would relieve the test set up code of uploading multipage.tiff on > each test run. > > We may even consider pre-installing the base db so that the switch-in > code can use db functions to clone it, rather than creating it from the > base dump for each test run that uses it. The latter could take a > significant amount of time, thereby severely slowing testing. > > >> After the test run completes, the testing application cleans up the > >> test run by requesting the deletion of the temporary resources and > >> the objectcache table entry associated with the test run. > > In some cases, tests will not change any data, e.g. testing dynamic > > skin elements in vector skin. Would it make sense not to tear down > > the testing environment in that case in order to save some time when > > testing repeatedly? I think, there is a conflict between performance > > and amount of data, but who wins? > > There may be ways to make things more efficient by not immediately > deleting the temporary resources. However, eventually we have to delete > them. So, there is a question of where the temporary resources > identifier is stored so it can be used later for a clean-up request. I > was assuming the switch-in request occurs in test suite start-up and > the clean-up request occurs in the test suite finishing code. But, we > should consider alternative implementation strategies. > > > In general, it seems to me that we have some similarity with what is > > called wiki family on mediawiki.org. One could see multiple testing > > environments as a set of multiple wikis that share a common codebase > > [1]. Does anybody have experience with wiki families and the object > > cache rsp. memcached? > > I agree. (In fact, I mentioned this previously). No need to reinvent > the wheel. > > > I am not sure whether we can use the same codebase as parsertests. > > I'd rather think, parser tests are a special case of what we are > > sketching here. On the other hand, I don't think it is a good idea to > > have two separate approaches for very similar tasks in the code. Do > > you think it would be feasible to separate the preparation part from > > both parser tests and selenium tests and build both of them on a > > common ground? > > > > > The parserTests code creates some temporary tables in an existing wiki > database. Currently, these tables are empty and prefixed with a static > identifier. The requirements for parserTests and selenium tests are > significantly different. While we may be able to learn from the > parserTests code, we would have to change the parserTest code > significantly in order to use it. I think it would actually be less > work to start from scratch. Of course, as you mention, there may be > code used to support wiki families that we could use without much > modification. > > -- > -- Dan Nessett > > > _______________________________________________ Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l >

On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett <dnessett(a)yahoo.com> wrote: I'd just treat it same as any other wiki. Whether you're running one or multiple wiki instances out of one copy of the code base, it just doesn't make a difference here. -- brion

On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote: Not to hammer this point to hard, but there is another reason for supporting switching in of temporary resources. We have one worked example of a Selenium test - PagedTiffHandler. During the course of its execution, it uploads some tiff files to test some of its functionality. Currently, once an image is uploded, there is no way to completely delete it (short of drastic administrative action involving raw database manipulation). So, running PagedTiffHandler twice doesn't work unless you switch in temporary resources on each run that are then deleted at the end of the run. -- -- Dan Nessett _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Given a test matrix with multiple OSes, this ain't something individual devs will be running in full over and over as they work. Assume automated batch runs, which can be distributed over as many databases and clients as you like. For small test subsets that are being used during testing the equation still doesn't change much: reset the wiki to known state, run the tests. Keep it simple! -- brion On Thursday, September 23, 2010, Dan Nessett <dnessett(a)yahoo.com> wrote: > On Wed, 22 Sep 2010 12:30:35 -0700, Brion Vibber wrote: > >> On Wed, Sep 22, 2010 at 11:09 AM, Dan Nessett <dnessett(a)yahoo.com> >> wrote: >> >>> Some have mentioned the possibility of using the wiki family logic to >>> help achieve these objectives. Do you have any thoughts on this? If >>> you think it is a good idea, how do we find out more about it? >>> >>> >> I'd just treat it same as any other wiki. Whether you're running one >> or multiple wiki instances out of one copy of the code base, it just >> doesn't make a difference here. >> >> -- brion > > Not to hammer this point to hard, but there is another reason for > supporting switching in of temporary resources. > > We have one worked example of a Selenium test - PagedTiffHandler. > During the course of its execution, it uploads some tiff files to test > some of its functionality. Currently, once an image is uploded, there > is no way to completely delete it (short of drastic administrative > action involving raw database manipulation). So, running > PagedTiffHandler twice doesn't work unless you switch in temporary > resources on each run that are then deleted at the end of the run. > > -- > -- Dan Nessett > > > _______________________________________________ Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l >

I am very much in favor of keeping it simple. I think the issue is whether we will support more than one regression test (or individual test associated with a regression test) running concurrently on the same test wiki. If not, then I agree, no switching logic is necessary.

After thinking about this some more I think you are right. We should at least start with something simple and only make it more complex (e.g., wiki resource switching) if the simple approach has significant problems. There is already a way to 'bundle' individual tests together. It is the selenium test suite. We could use that. We could then break up a regression test into separate test suites that could run concurrently. Summarizing, here is my understanding of your proposal:

There are still some open questions: + How do you establish a fresh wiki for a URL used previously for a test run?

any one of these. How do you synchronize regression test initiation so there is some sort of lock on a test wiki currently running a regression test?

Not sure I get this. Here is what I understand would happen when a developer checks in a revision: + A script runs that manages the various regression tests run on the revision (e.g., parserTests, PHPUnit tests, the Selenium-based regression test). + The Selenium regression test needs a URL to work with. There are a fixed set of these defined in httpd.conf.

requirement that it isn't currently busy running a regression test for a different revision. So, you need resource access control on the URLs.

On Thu, 23 Sep 2010 14:41:32 -0700, Brion Vibber wrote: Are there instructions somewhere that describe how to do this?

Thinking about this a bit, we seem to have come full circle. If we use a URL per regression test run, then we need to multiplex wiki resources. When you set up a wiki family, the resources are permanent. But, for a test run, you need to set them up, use them and then reclaim them. The resources are the db, the images directory, cache data, etc.

I appreciate your recent help, so I am going to ignore the tone of your last message and focus on issues. While a test run can set up, use and then delete the temporary resources it needs (i.e., db, images directory, etc.), you really haven't answered the question I posed. If the test run ends abnormally, then it will not delete those resources. There has to be a way to garbage collect orphaned dbs, images directories and cache entries.

My personal view is we should start out simple (as you originally suggested) with a set of fixed URLs that are used serially by test runs. Implementing this is probably the easiest option and would allow us to get something up and running quickly. This approach doesn't require significant development, although it does require a way to control access to the URLs so test runs don't step on each other.

(If using memcached, be sure to clear those out, reinitialize, or otherwise do something that forces old values to be cleared or ignored.)

Here is all that is required: * a single wildcard entry in Apache configuration * one or two lines in LocalSettings.php to pull a DB name from the hostname/path/CLI parameters. As for cleaning up resources to keep the machine from getting clogged, it's very unlikely that your test wikis will fill up a multi-hundred-gigabyte drive in the middle of a run. If you find that they do, there's still no need to tie cleanup of any particular run to any particular other run. All you need to know is which runs have completed and can now be cleaned up.

Here is all that is required: * a single wildcard entry in Apache configuration * one or two lines in LocalSettings.php to pull a DB name from the hostname/path/CLI parameters. As for cleaning up resources to keep the machine from getting clogged, it's very unlikely that your test wikis will fill up a multi-hundred-gigabyte drive in the middle of a run. If you find that they do, there's still no need to tie cleanup of any particular run to any particular other run. All you need to know is which runs have completed and can now be cleaned up.