Hello everybody,
in order to further develop the selenium framework [1], I need to make a few design decisions, especially on coding conventions, which I'd like to discuss on this list, since they affect the way of how extension- and core developers write their tests.
1) Where are the tests located? I suggest for core to put them into maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is <extensiondir>/tests/selenium.
2) How are the tests organized? Tests are organized in testing suites. Each suite represents a conhesive set of tests. So it is possible to have more than one test suite per extension / core area. Test suites are technically classes. The files should follow this naming convention: <NameOfExtension><[Subset]>TestSuite.php. The subset is optional. For example, in the PagedTiffHandler extension, it would be PagedTiffHandlerTestSuite.php and PagedTiffHandlerUploadsTestSuite.php. This should also be the name of the class. Alternatively, we could use the word "Selenium" somewhere in there in order to be able to divide between unit and selenium tests. In that case I suggest to use PagedTiffHandlerSeleniumTestSuite.php and PagedTiffHandlerUploadsSeleniumTestSuite.php. Hmmm... this gives pretty long names. Any suggestions?
3) How does the framework know there are tests? The tests should be registered with the autoloader in the extension entry file. In core, they should be registered directly with the autoloader.
4) Which tests should be executed? Since Selenium tests are slow, not every test should be executed in each test run. So At the moment, there is a variable $wgSeleniumTestSuites which can be set in LocalSettings.php and which contains the tests that should be run. If things become more dynamically (e.g. when tests should be run on svn commit), there could be a function to add to this variable.
5) Aesthetics... There is an awful lot of "Selenium" in the class names, method names, file names and variable names. It might be a good idea to use "Sn" everywhere except for path names.
Two things need to be kept in mind: * The idea is to use a similar structure for unit- and selenium tests (selenium tests are based on unit tests anyway). I assume at some point, the tests should also be compatible with a continuous integration server. * The wiki that executes the selenium tests is not neccesarily the one that is being tested if the tests run against a selenium grid.
If anybody would like to share their opinion on my suggestions, I'd be very glad!
Regards, Markus
[1] http://www.mediawiki.org/wiki/SeleniumFramework (documentation will be updated soon..)
On 08/05/2010 12:37:01 PM, Markus Glaser - glaser@hallowelt.biz wrote:
Hello everybody,
snip....
- Aesthetics... There is an awful lot of "Selenium" in the class names, method names, file names and variable names. It might be a good idea to use "Sn" everywhere except for path names.
Shortening the name is a good idea. Since Selenium is a chemical element, I think would be less confusing if you used the element abbreviation.
Sn is Tin (stannum), Se is Selenium.
Jim Laurino
snip...
Markus Glaser glaser@hallowelt.biz writes:
- Where are the tests located? I suggest for core to put them into
maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is <extensiondir>/tests/selenium.
Sounds fine.
In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that <extensiondir>/tests will contain tests that should be run using PHPUnit.
Alternatively, we could use the word "Selenium" somewhere in there in order to be able to divide between unit and selenium tests.
I think putting them in the selenium directory (or the “Se” directory) is sufficient.
- How does the framework know there are tests?
Can I suggest that the framework can see that an extension has tests simply by the presence of the <extensiondir>/tests directory containing a <Extension>*TestSuite.php file?
The <extensiondir>/tests/<Extension>TestSuite.php file should define a class using the name <Extension>TestSuite which has a static method suite(). See the PHPUnit documentation at http://bit.ly/b9L50r for how this is set up.
This static suite() method should take care of letting the autoloader know about any test classes so the test classes are only available during testing.
So, for your example using PagedTiffHandler, there would be the files:
PagedTiffHandler/tests/PagedTiffHandlerTestSuite.php PagedTiffHandler/tests/PagedTiffHandlerUploadsTestSuite.php
- Which tests should be executed?
By default all the test suites in <extensiondir>/tests should be run.
It is should be possible to specify which particular test to run by using whatever command line arguments to the CLI.
This seems better to me than defining a new global. If some tests should only be run rarely, that information can be put in the TestSuite class for te extension.
In this way, I think it is possible to remove all the $wgSelenium* variables from the DefaultSettings.php file.
(I plan to do something similar with the $wgParserTest* variables as well — these sorts of things don't seem like they belong in Core.)
Mark.
On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershberger mah@everybody.org wrote:
Markus Glaser glaser@hallowelt.biz writes:
- Where are the tests located? I suggest for core to put them into
maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is <extensiondir>/tests/selenium.
Sounds fine.
In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that <extensiondir>/tests will contain tests that should be run using PHPUnit.
I would prefer moving them to a subdirectory of /tests/. As we hopefully amass more unit tests, keeping them in the top-level will get a bit confusing when trying to distinguish them from supporting code (shared setUp and tearDown code, the bootstrap stuff, etc)
Something like /maintenance/tests/unit/ to mirror /maintenance/tests/ selenium/ would make the most sense.
Consistency and thinking ahead is nice :)
-Chad
I think /tests/unit and /tests/acceptance would be reasonable places to put things, and if they are both within maintenance or in the root doesn't really matter to me.
Remember Selenium is a framework for doing acceptance testing, not unit testing. I don't quite see the purpose of specifying the framework name in our directory structure. Are we planning on using more than one unit or acceptance testing framework?
My 2 cents.
- Trevor
On 8/5/10 3:55 PM, Chad wrote:
On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershbergermah@everybody.org wrote:
Markus Glaserglaser@hallowelt.biz writes:
- Where are the tests located? I suggest for core to put them into
maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is<extensiondir>/tests/selenium.
Sounds fine.
In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that<extensiondir>/tests will contain tests that should be run using PHPUnit.
I would prefer moving them to a subdirectory of /tests/. As we hopefully amass more unit tests, keeping them in the top-level will get a bit confusing when trying to distinguish them from supporting code (shared setUp and tearDown code, the bootstrap stuff, etc)
Something like /maintenance/tests/unit/ to mirror /maintenance/tests/ selenium/ would make the most sense.
Consistency and thinking ahead is nice :)
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hello,
You are right that having "unit" and "selenium" folders in tests does not quite fit.
However, "acceptance" I don't find appropriate, either.
IMHO, Selenium actually is a framework for system testing (as it evaluates the system functionalities from a user perspective). So, how about having a folder "system"?
Regards,
Benedikt
-- Karlsruhe Institute of Technology (KIT) Institute of Applied Informatics and Formal Description Methods (AIFB)
Benedikt Kämpgen Research Associate
Kaiserstraße 12 Building 11.40 76131 Karlsruhe, Germany
Phone: +49 721 608-7946 Fax: +49 721 608-6580 Email: benedikt.kaempgen@kit.edu Web: http://www.kit.edu/
KIT - University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Trevor Parscal Sent: Friday, August 06, 2010 8:27 PM To: wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
I think /tests/unit and /tests/acceptance would be reasonable places to put things, and if they are both within maintenance or in the root doesn't really matter to me.
Remember Selenium is a framework for doing acceptance testing, not unit testing. I don't quite see the purpose of specifying the framework name in our directory structure. Are we planning on using more than one unit or acceptance testing framework?
My 2 cents.
- Trevor
On 8/5/10 3:55 PM, Chad wrote:
On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershbergermah@everybody.org wrote:
Markus Glaserglaser@hallowelt.biz writes:
- Where are the tests located? I suggest for core to put them into
maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is<extensiondir>/tests/selenium.
Sounds fine.
In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that<extensiondir>/tests will contain tests that should be run using PHPUnit.
I would prefer moving them to a subdirectory of /tests/. As we hopefully amass more unit tests, keeping them in the top-level will get a bit confusing when trying to distinguish them from supporting code (shared setUp and tearDown code, the bootstrap stuff, etc)
Something like /maintenance/tests/unit/ to mirror /maintenance/tests/ selenium/ would make the most sense.
Consistency and thinking ahead is nice :)
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 8/7/10 9:58 AM, Benedikt Kaempgen wrote:
Hello,
You are right that having "unit" and "selenium" folders in tests does not quite fit.
However, "acceptance" I don't find appropriate, either.
IMHO, Selenium actually is a framework for system testing (as it evaluates the system functionalities from a user perspective). So, how about having a folder "system"?
Regards,
Benedikt
-- Karlsruhe Institute of Technology (KIT) Institute of Applied Informatics and Formal Description Methods (AIFB)
Benedikt Kämpgen Research Associate
Kaiserstraße 12 Building 11.40 76131 Karlsruhe, Germany
Phone: +49 721 608-7946 Fax: +49 721 608-6580 Email: benedikt.kaempgen@kit.edu Web: http://www.kit.edu/
KIT - University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association
-----Original Message----- From: wikitech-l-bounces@lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Trevor Parscal Sent: Friday, August 06, 2010 8:27 PM To: wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Testing Framework (was Selenium Framework - Question on coding conventions)
I think /tests/unit and /tests/acceptance would be reasonable places to put things, and if they are both within maintenance or in the root doesn't really matter to me.
Remember Selenium is a framework for doing acceptance testing, not unit testing. I don't quite see the purpose of specifying the framework name in our directory structure. Are we planning on using more than one unit or acceptance testing framework?
My 2 cents.
- Trevor
On 8/5/10 3:55 PM, Chad wrote:
On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershbergermah@everybody.org wrote:
Markus Glaserglaser@hallowelt.biz writes:
- Where are the tests located? I suggest for core to put them into
maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is<extensiondir>/tests/selenium.
Sounds fine.
In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that<extensiondir>/tests will contain tests that should be run using PHPUnit.
I would prefer moving them to a subdirectory of /tests/. As we hopefully amass more unit tests, keeping them in the top-level will get a bit confusing when trying to distinguish them from supporting code (shared setUp and tearDown code, the bootstrap stuff, etc)
Something like /maintenance/tests/unit/ to mirror /maintenance/tests/ selenium/ would make the most sense.
Consistency and thinking ahead is nice :)
-Chad
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I'm not /totally/ opposed to breaking away from the standard terminology of unit/integration/acceptance testing... We could call it something more descriptive than system though - perhaps "client"...
- Trevor
On Sat, Aug 7, 2010 at 3:35 PM, Trevor Parscal tparscal@wikimedia.orgwrote:
I'm not /totally/ opposed to breaking away from the standard terminology of unit/integration/acceptance testing... We could call it something more descriptive than system though - perhaps "client"...
Going with names like "unit" and "client" have the advantage that the names document what type of tests that they are, with the framework not really being important.
However, names like "selenium" and "phpunit" have the advantage that, should we decide to start moving to a new framework (say, the fictional "foounit"), we put the new foounit tests in a foounit directory, keep the phpunit tests in the phpunit directory, and there's never a confusing mixing of tests from different frameworks in the same directory.
It's not that big of a deal either way. My 2c: pick something reasonably consistent, then put a very clear README.txt file that describes what is in the tests/ directory. That way, if we decide to use the framework name, and if someone doesn't know that "selenium" is for system/client/acceptance/whateveryouwanttocallit testing, they can look at the README and figure it out.
Rob
Trevor Parscal tparscal@wikimedia.org writes:
Remember Selenium is a framework for doing acceptance testing, not unit testing.
Right. They can also serve as (I hope) regression tests — which is what I'm mostly concerned with here.
Mark.
On Fri, Aug 6, 2010 at 2:27 PM, Trevor Parscal tparscal@wikimedia.org wrote:
I think /tests/unit and /tests/acceptance would be reasonable places to put things, and if they are both within maintenance or in the root doesn't really matter to me.
Remember Selenium is a framework for doing acceptance testing, not unit testing. I don't quite see the purpose of specifying the framework name in our directory structure. Are we planning on using more than one unit or acceptance testing framework?
I think it's best to pick whatever term will be the most intelligible. The idea of unit tests is very widely known, so tests/unit/ should be readily understood by anyone who knows much about automated software testing at all. Would tests/selenium/ or tests/acceptance/ be more comprehensible to the average person? I had never heard the term "acceptance test" before now, and the Wikipedia article on it makes it sound like a pretty vague term.
Either way, we should probably throw parser tests in the same directory.
On Thu, 05 Aug 2010 18:47:58 -0400, Mark A. Hershberger wrote:
Markus Glaser glaser@hallowelt.biz writes:
- Where are the tests located? I suggest for core to put them into
maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is <extensiondir>/tests/selenium.
Sounds fine.
In the same way, since maintenance/tests contains tests that should be run using PHPUnit, we can say that <extensiondir>/tests will contain tests that should be run using PHPUnit.
Alternatively, we could use the word "Selenium" somewhere in there in order to be able to divide between unit and selenium tests.
I think putting them in the selenium directory (or the “Se” directory) is sufficient.
- How does the framework know there are tests?
Can I suggest that the framework can see that an extension has tests simply by the presence of the <extensiondir>/tests directory containing a <Extension>*TestSuite.php file?
The <extensiondir>/tests/<Extension>TestSuite.php file should define a class using the name <Extension>TestSuite which has a static method suite(). See the PHPUnit documentation at http://bit.ly/b9L50r for how this is set up.
This static suite() method should take care of letting the autoloader know about any test classes so the test classes are only available during testing.
So, for your example using PagedTiffHandler, there would be the files:
PagedTiffHandler/tests/PagedTiffHandlerTestSuite.php PagedTiffHandler/tests/PagedTiffHandlerUploadsTestSuite.php
- Which tests should be executed?
By default all the test suites in <extensiondir>/tests should be run.
It is should be possible to specify which particular test to run by using whatever command line arguments to the CLI.
This seems better to me than defining a new global. If some tests should only be run rarely, that information can be put in the TestSuite class for te extension.
In this way, I think it is possible to remove all the $wgSelenium* variables from the DefaultSettings.php file.
(I plan to do something similar with the $wgParserTest* variables as well — these sorts of things don't seem like they belong in Core.)
Mark.
I don't think walking through all the extensions looking for test subdirectories and then running all tests therein is a good idea. First, in a large installation with many extensions, this takes time and delays the test execution. Second, if a developer is working on a particular extension or part of the core, it will be common to run the tests associated with that for regression purposes. Making the developer specify the extension or core tests to run on the RunSeleniumTests command line is irritating (at least, it would irritate me). Specifying the test suite(s) to be run in LocalSettings.php is a set and forget approach that allows the developer to get on with the work.
However, I do agree that the number of global variables associated with the selenium framework is getting large and has the potential of growing over time. One solution is to use a multi-dimensional associative array (much like $wgGroupPermissions). We could use a global variable $wgSelenium and move all selenium framework values into it. For example:
$wgSelenium['wiki']['host'] = 'localhost'; $wgSelenium['wiki']['wikiurl'] = false; $wgSelenium['wiki']['loginname'] = 'Wikiuser; $wgSelenium['wiki']['password'] = ''; $wgSelenium['server']['port'] = 4444;
etc.
The only global we may wish to keep separate is $wgEnableSelenium, since it specifies whether $wgSelenium is used.
Dan Nessett dnessett@yahoo.com writes:
I don't think walking through all the extensions looking for test subdirectories and then running all tests therein is a good idea. First, in a large installation with many extensions, this takes time and delays the test execution.
Globbing for extensions/*/tests/TestSettings.php doesn't take long at all.
However I am looking at a way to test extensions independently of installation.
This means I can't depend on hooks or global variables, so I need another way to find out if an extension has tests available.
Making the developer specify the extension or core tests to run on the RunSeleniumTests command line is irritating (at least, it would irritate me)
No doubt. So why not allow per-user files to set this instead of using LocalSettings.php?
Mark.
Mark A. Hershberger wrote:
Dan Nessett dnessett@yahoo.com writes:
I don't think walking through all the extensions looking for test subdirectories and then running all tests therein is a good idea. First, in a large installation with many extensions, this takes time and delays the test execution.
Globbing for extensions/*/tests/TestSettings.php doesn't take long at all.
However I am looking at a way to test extensions independently of installation.
This means I can't depend on hooks or global variables, so I need another way to find out if an extension has tests available.
You could have a global variable and also set the convention of storing them in extensions/*/tests/TestSettings.php
Anyway, I don't think you could do that. If the extension isn't installed, you won't pass whatever extension behavior it is testing, would it?
On Sun, Aug 8, 2010 at 6:47 PM, Platonides Platonides@gmail.com wrote:
Anyway, I don't think you could do that. If the extension isn't installed, you won't pass whatever extension behavior it is testing, would it?
It should be possible to run unit tests in some cases even if the extension isn't actually installed.
On Sat, 07 Aug 2010 23:30:16 -0400, Mark A. Hershberger wrote:
Dan Nessett dnessett@yahoo.com writes:
I don't think walking through all the extensions looking for test subdirectories and then running all tests therein is a good idea. First, in a large installation with many extensions, this takes time and delays the test execution.
Globbing for extensions/*/tests/TestSettings.php doesn't take long at all.
However I am looking at a way to test extensions independently of installation.
This means I can't depend on hooks or global variables, so I need another way to find out if an extension has tests available.
Making the developer specify the extension or core tests to run on the RunSeleniumTests command line is irritating (at least, it would irritate me)
No doubt. So why not allow per-user files to set this instead of using LocalSettings.php?
Mark.
Testing uninstalled extensions may make sense for unit tests, but not for selenium tests. The latter exercise the code through a browser, so the extension must be installed for selenium testing.
I'm not sure what are the advantages of "per-user" configuration files. For unit tests the tester directly accesses the code and so has direct access to LocalSettings. For selenium testing, we originally had a configuration file called SeleniumLocalSettings.php, but that was abandoned in favor of putting the configuration information in DefaultSettings and LocalSettings.
As stated previously, selenium tests exercise MW code by accessing the wiki through a browser. I don't see how a 'per-user' configuration file would be integrated without introducing some serious security issues.
Dan Nessett dnessett@yahoo.com writes:
For unit tests the tester directly accesses the code and so has direct access to LocalSettings. For selenium testing, we originally had a configuration file called SeleniumLocalSettings.php, but that was abandoned in favor of putting the configuration information in DefaultSettings and LocalSettings.
I would have to go look at why that decision was made, but it seems like it just allows the code to be polluted with Selenium-specific identifiers that production code should not have *anything* to do with.
I would like to keep testing-specific code (and variables) separate and distinct from production code.
As stated previously, selenium tests exercise MW code by accessing the wiki through a browser.
Agreed.
Mark.
Embrace Ignorance. Just don't get too attached.
On 8/9/10 11:27 AM, Mark A. Hershberger wrote:
Dan Nessettdnessett@yahoo.com writes:
For unit tests the tester directly accesses the code and so has direct access to LocalSettings. For selenium testing, we originally had a configuration file called SeleniumLocalSettings.php, but that was abandoned in favor of putting the configuration information in DefaultSettings and LocalSettings.
I would have to go look at why that decision was made, but it seems like it just allows the code to be polluted with Selenium-specific identifiers that production code should not have *anything* to do with.
I would like to keep testing-specific code (and variables) separate and distinct from production code.
As stated previously, selenium tests exercise MW code by accessing the wiki through a browser.
Agreed.
Mark.
Embrace Ignorance. Just don't get too attached.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I don't know where this landed, but I wanted to point out that system testing might be a better name for out use of Selenium, Acceptance testing has more of a "customer is accepting a product" connotation.
http://en.wikipedia.org/wiki/System_testing
- Trevor
Trevor Parscal tparscal@wikimedia.org writes:
I don't know where this landed, but I wanted to point out that system testing might be a better name for out use of Selenium, Acceptance testing has more of a "customer is accepting a product" connotation.
During our discussion last Friday, Marcus, Priyanka and I decided to stick with the name of the framework (in this case, “Selenium”) since there could be other tools used for similar sorts of testing.
In other words, if we named the directory “system” and then had system tests performed with a tool other than Selenium, we'd be back to square one — that is “Where do we put the tests and What do we name them?”
Mark.
On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershberger mah@everybody.org wrote:
Can I suggest that the framework can see that an extension has tests simply by the presence of the <extensiondir>/tests directory containing a <Extension>*TestSuite.php file?
IMO, the way parser tests do it is smarter. When you install the extension, it adds the location of the test files to $wgParserTestFiles. That way, only the tests associated with installed extensions will run. If you want to force only particular tests to run all the time, you can also modify the variable in LocalSettings.php as usual.
On Fri, 06 Aug 2010 12:38:39 -0400, Aryeh Gregor wrote:
On Thu, Aug 5, 2010 at 6:47 PM, Mark A. Hershberger mah@everybody.org wrote:
Can I suggest that the framework can see that an extension has tests simply by the presence of the <extensiondir>/tests directory containing a <Extension>*TestSuite.php file?
IMO, the way parser tests do it is smarter. When you install the extension, it adds the location of the test files to $wgParserTestFiles. That way, only the tests associated with installed extensions will run. If you want to force only particular tests to run all the time, you can also modify the variable in LocalSettings.php as usual.
We are doing something similar. In the extension require() file, the test suite is added to $wgAutoloadClasses. Right now the entry in $wgSeleniumTestSuites is pushed in LocalSettings. However, we could establish the convention that it is pushed in the extension require() file as well. Then all extensions with test suites would automatically load them. To tailor this, the entries in $wgSeleniumTestSuites could be removed in LocalSettings.
On 05.08.2010, 20:37 Markus wrote:
- Where are the tests located? I suggest for core to put them into
maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is <extensiondir>/tests/selenium.
- How are the tests organized? Tests are organized in testing
suites. Each suite represents a conhesive set of tests. So it is possible to have more than one test suite per extension / core area. Test suites are technically classes. The files should follow this naming convention: <NameOfExtension><[Subset]>TestSuite.php. The subset is optional. For example, in the PagedTiffHandler extension, it would be PagedTiffHandlerTestSuite.php and PagedTiffHandlerUploadsTestSuite.php. This should also be the name of the class. Alternatively, we could use the word "Selenium" somewhere in there in order to be able to divide between unit and selenium tests. In that case I suggest to use PagedTiffHandlerSeleniumTestSuite.php and PagedTiffHandlerUploadsSeleniumTestSuite.php. Hmmm... this gives pretty long names. Any suggestions?
- How does the framework know there are tests? The tests should be
registered with the autoloader in the extension entry file. In core, they should be registered directly with the autoloader.
I've already implemented a system allowing extensions to register their PHPUnit tests with UnitTestsList hook, and it's much more simple: a couple of lines on extension's side and one wfRunHooks() call in the test runner.
On 8/5/10 9:37 AM, Markus Glaser wrote:
Hello everybody,
in order to further develop the selenium framework [1], I need to make a few design decisions, especially on coding conventions, which I'd like to discuss on this list, since they affect the way of how extension- and core developers write their tests.
Where are the tests located? I suggest for core to put them into maintenance/tests/selenium. That is where they are now. For extensions I propse a similar structure, that is<extensiondir>/tests/selenium.
How are the tests organized? Tests are organized in testing suites. Each suite represents a conhesive set of tests. So it is possible to have more than one test suite per extension / core area. Test suites are technically classes. The files should follow this naming convention:<NameOfExtension><[Subset]>TestSuite.php. The subset is optional. For example, in the PagedTiffHandler extension, it would be PagedTiffHandlerTestSuite.php and PagedTiffHandlerUploadsTestSuite.php. This should also be the name of the class. Alternatively, we could use the word "Selenium" somewhere in there in order to be able to divide between unit and selenium tests. In that case I suggest to use PagedTiffHandlerSeleniumTestSuite.php and PagedTiffHandlerUploadsSeleniumTestSuite.php. Hmmm... this gives pretty long names. Any suggestions?
In PHPUnit a TestSuite is a class is a class that orchestrates the execution of many Test classes, so PagedTiffHandlerTest would exercise the PagedTiffHandler class, but ImagePagingHandlersTestSuite might conduct tests for PagedTiffHandler and also PdfHandler or whatever.
- How does the framework know there are tests? The tests should be registered with the autoloader in the extension entry file. In core, they should be registered directly with the autoloader.
PHPUnit uses either a TestSuite class to determine what to run, or an XML file. Explicitly listing each test is not neesecary to run all tests, but TestSuite classes or XML defined suites help run a portion of tests. I don't know if Selenium supports this kind of functionality, but it would be nice if we could find a similar approach between the two frameworks.
Which tests should be executed? Since Selenium tests are slow, not every test should be executed in each test run. So At the moment, there is a variable $wgSeleniumTestSuites which can be set in LocalSettings.php and which contains the tests that should be run. If things become more dynamically (e.g. when tests should be run on svn commit), there could be a function to add to this variable.
Aesthetics... There is an awful lot of "Selenium" in the class names, method names, file names and variable names. It might be a good idea to use "Sn" everywhere except for path names.
PHP 5.3 namespaces? (with backslashes! -- evil music --)
Two things need to be kept in mind:
- The idea is to use a similar structure for unit- and selenium tests (selenium tests are based on unit tests anyway). I assume at some point, the tests should also be compatible with a continuous integration server.
- The wiki that executes the selenium tests is not neccesarily the one that is being tested if the tests run against a selenium grid.
If anybody would like to share their opinion on my suggestions, I'd be very glad!
Regards, Markus
[1] http://www.mediawiki.org/wiki/SeleniumFramework (documentation will be updated soon..) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org