On Fri, 17 Sep 2010 19:13:33 +0000, Dan Nessett wrote:
On Fri, 17 Sep 2010 18:40:53 +0000, Dan Nessett
wrote:
I have been tasked to evaluate whether we can use
the parserTests db
code for the selenium framework. I just looked it over and have serious
reservations. I would appreciate any comments on the following
analysis.
The environment for selenium tests is different than that for
parserTests. It is envisioned that multiple concurrent tests could run
using the same MW code base. Consequently, each test run must:
+ Use a db that if written to will not destroy other test wiki
information.
+ Switch in a new images and math directory so any writes do not
interfere with other tests.
+ Maintain the integrity of the cache.
Note that tests would *never* run on a production wiki (it may be
possible to do so if they do no writes, but safety considerations
suggest they should always run on a test data, not production data). In
fact production wikis should always retain the setting
$wgEnableSelenium = false, to ensure selenium test are disabled.
Given this background, consider the following (and feel free to comment
on it):
parserTests temporary table code:
A fixed set of tables are specified in the code. parserTests creates
temporary tables with the same name, but using a different static
prefix. These tables are used for the parserTests run.
Problems using this approach for selenium tests:
+ Selenium tests on extensions may require use of extension specific
tables, the names of which cannot be elaborated in the code.
+ Concurrent test runs of parserTests are not supported, since the
temporary tables have fixed names and therefore concurrent writes to
them by parallel test runs would cause interference.
+ Clean up from aborted runs requires dropping fossil tables. But, if a
previous run tested an extension with extension-specific tables, there
is no way for a test of some other functionality to figure out which
tables to drop.
For these reasons, I don't think we can reuse the parserTests code.
However, I am open to arguments to the contrary.
After reflection, here are some other problems.
+ Some tests assume the existence of data in the db. For example, the
PagedTiffHandler tests assume the image Multipage.tiff is already
loaded. However, this requires an entry in the image table. You could
modify the test to clone the existing image table, but that means you
have problems with:
+ Some tests assume certain data is *not* in the db. PagedTiffHandler
has tests that upload images. These cannot already be in the images
table. So, you can't simply clone the images table.
All of this suggests to me that a better strategy is:
+ When the test run begins, clone a db associated with the test suite.
+ Switch the wiki to use this db and return a cookie or some other state
information that identifies this test run configuration.
+ When the test suite runs, each wiki access supplies this state so the
wiki code can switch in the correct db.
+ Cleanup of test runs requires removing the cloned db.
+ To handled aborted runs, there needs to be a mechanism to time out
cloned dbs and the state associated with the test run.
Regardless of how we implement the persistent storage for managing test
runs, there needs to be a way to trigger it use. To minimize the changes
to core, we need a hook that runs after processing LocalSettings (and by
implication DefaultSettings), but before any wiki state is accessed
(e.g., before accessing the db, the images directory, any cached data). I
looked at the existing hooks, but so far have not found one that appears
suitable.
So, either we need to identify an appropriate existing hook, or we need
to add a hook that meets the requirements.
--
-- Dan Nessett