On Thu, 23 Sep 2010 10:29:58 -0700, Brion Vibber wrote:
On Thu, Sep 23, 2010 at 9:46 AM, Dan Nessett
<dnessett(a)yahoo.com> wrote:
I am very much in favor of keeping it simple. I
think the issue is
whether we will support more than one regression test (or individual
test associated with a regression test) running concurrently on the
same test wiki. If not, then I agree, no switching logic is necessary.
*nod*
It might be a good idea to divide the test set up into, say 'channels'
or 'bundles' which are independent of each other, but whose individual
steps must run in sequence. If the tests are designed well, you should
be able to run tests from multiple 'channels' on the same wiki
simultaneously -- just as in the real world, multiple users are doing
multiple things on your wiki at the same time.
So one test set might be:
* create page A-{{unique-id}} as user One-{{unique-id}} * open editing
page as user One-{{unique-id}} * open and save the page as user
Two-{{unique-id}} * save the page as user One-{{unique-id}} * confirm
edit conflict / merging behavior was as expected
And another might be:
* register a new user account User--{{unique-id}} * change skin option
in preferences
* confirm that the skin changed as expected
These tests don't interfere with each other -- indeed if they did, that
would be information you'd need to know about a serious bug!
Most test sets should be fairly separate like this; only some that
change global state (say, a site administrator using a global
configuration panel to change the default skin) would need to be run
separately.
-- brion
After thinking about this some more I think you are right. We should at
least start with something simple and only make it more complex (e.g.,
wiki resource switching) if the simple approach has significant problems.
There is already a way to 'bundle' individual tests together. It is the
selenium test suite. We could use that. We could then break up a
regression test into separate test suites that could run concurrently.
Summarizing, here is my understanding of your proposal:
+ A regression test run comprises a set of test suites, each of which may
run concurrently.
+ If you want to run multiple regression tests concurrently, use
different test wikis (which can run over the same code base, but which
are identified by different URLs - i.e., rely on httpd to multiplex
multiple concurrent regression tests).
+ If you want to run parts of a regression test concurrently, the unit of
concurrency is the test suite.
+ A regression test begins by establishing a fresh wiki. Each test suite
starts by establishing the wiki state it requires (e.g., for
PagedTiffHanlder, loading Multipage.tiff).
+ It is an open question whether test suite or total regression test
cleanup is necessary. It may be possible to elide this step and simply
rely on regression test initialization to cleanup any wiki state left
around by a previous test run.
There are still some open questions:
+ How do you establish a fresh wiki for a URL used previously for a test
run?
+ URLs identify test wikis. Only one regression test can run at time on
any one of these. How do you synchronize regression test initiation so
there is some sort of lock on a test wiki currently running a regression
test?
--
-- Dan Nessett