Hi Gerard,
I am very interested in the tool you mention. Let's keep the discussion on list, since I suspect there are others who might want to set up a regression test environment either now or later.
Can you provide some pointers how to use this tool? Is it described in the standard SVN documentation or is that located somewhere else? What is its name? Would its use allow testing against the most recent version in trunk?
My initial thoughts for such a regression test installation are:
I think some extensions involve database schema changes. My initial idea is to create a new installation, make all of the schema changes necessary for any extensions in the regression test set and then dump the database. This dump could then be used to set up new regression test installations (or reinitialize existing test installations). Perhaps the dump could be placed in Subversion in a "test" area.
Just loading a set of extensions in an installation doesn't really provide much in the way of verifying they work together. The test needs to use them together. One way to do this would be to create a set of pages that concurrently access the extensions when the pages are rendered. Knowing which extensions to use together is a key to this approach. Perhaps there is information in Bugzilla that would help determine that. Also, extension authors might provide some tests that could be incorporated into the test set. If you or others have some ideas on how to test extensions that would be very helpful.
When I worked on Solaris at Sun in the mid-90s, developers were required to regression test their changes before submitting them (through a gatekeeper) for inclusion in the nightly build. Those who failed to do so and broke the build had their hands slapped. Perhaps something similar might be established for the mediawiki development process. Extension authors might be required to: 1) provide some extension tests that could included in the regression test set (if their extensions ever become important enough to do that), and 2) run their extension tests and the standard tests against a standard regression test installation and provide evidence that there are no problems before their extensions are included in the mediawiki extensions matrix.
Dan
--- On Wed, 7/8/09, Gerard Meijssen gerard.meijssen@gmail.com wrote:
From: Gerard Meijssen gerard.meijssen@gmail.com Subject: Re: [Wikitech-l] Defining a configuration for regression testing To: "Wikimedia developers" wikitech-l@lists.wikimedia.org Cc: "Kim Bruning" kim@bruning.xs4all.nl Date: Wednesday, July 8, 2009, 10:28 PM Hoi. In Subversion there is a tool created for the setup of environments. What it does well is setup an environment with a specific configuration. This configuration for a specific environment can be found in a file. In this way it is possible to define a specific revision or tag for either MediaWiki itself or for an extension. The software is such that you can specify multiple languages for an environment.. As there are important differences because of the language, the script you want to be able to test for a key subset of wikis. Duplication of an initial created wiki works well.
When you look at ALL the extensions used whereever on WMF projects, you will find that they are not an homongous bunch; they are not used together. This means that you may want to have multiple environments configured. At this time there is a Wikipeida configuration and a Usability Initiative configuration. Given that the configuration is in a file, there is room to indicate a specific revision..
As you can imagine, there are scripts to install particular extensions that can not be installed in a default way.
When you have an interest, contact me or ask on this list. Thanks, GerardM
2009/7/9 dan nessett dnessett@yahoo.com
I am setting up a testing environment for mediawiki
and the first thing
that came to mind is testing new extensions against a
"regression test
configuration". That raises the question of what
should constitute such a
configuration. One issue is which extensions should be
loaded.
There are over 2000 extensions in the mediawiki
extensions matrix and 512
stable extensions. It would be impractical to run a
configuration with all
of either class. So, I asked around and received a
suggestion that at the
very least the extensions on the wikimedia servers
should be loaded. I went
to http://noc.wikimedia.org/conf/ and copied
CommonSettings.php. From it
I extracted 75 extensions that are used on wikimedia's
servers. I list these
below.
A question for readers of this list is: should a
regression test
configuration load only these extensions or should it
load others? Another
question is: what other settings should define a
regression test
configuration.
Wikimedia installed extensions:
Timeline, wikihiero, SiteMatrix, CharInsert,
CheckUser,
SpecialMakesysop, Makebot, ParserFunctions, Cite,
InputBox,
ExpandTemplates, ImageMap, SyntaxHighlight_GeSHi,
DoubleWiki, Poem,
PovWatch, AjaxTest, UnicodeConverter, CategoryTree,
ProofreadPage, lst,
SpamBlacklist, UploadBlacklist, TitleBlacklist, Quiz,
Gadgets,
OggHandler, AssertEdit, FormPreloadPostCache,
SkinPerPage, Schulenburg,
Tomas, ContributionReporting, ContributionTracking,
ContactPage,
ExtensionDistributor, GlobalBlocking, TrustedXFF,
ContactPage,
SecurePoll, OAIRepo, DynamicPageList, Nogomatch, SpecialCrossNamespaceLinks, SpecialRenameuser,
SpecialNuke, AntiBot,
TorBlock, CookieBlock, ScanSet, SpecialCite,
FixedImage, UserThrottle,
ConfirmEdit, FancyCaptcha, HideRevision, AntiSpoof,
CentralAuth,
DismissableSiteNotice, UsernameBlacklist,
MiniDonation, CentralNotice,
TitleKey, WikimediaMessages, SimpleAntiSpam,
Collection, NewUserMessage,
CodeReview, Drafts, Configure, AbuseFilter,
ClientSide, CommunityVoice,
PdfHandler, UsabilityInitiative
Regards,
Dan Nessett
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Jul 9, 2009 at 11:39 AM, dan nessettdnessett@yahoo.com wrote:
When I worked on Solaris at Sun in the mid-90s, developers were required to regression test their changes before submitting them (through a gatekeeper) for inclusion in the nightly build. Those who failed to do so and broke the build had their hands slapped. Perhaps something similar might be established for the mediawiki development process. Extension authors might be required to: 1) provide some extension tests that could included in the regression test set (if their extensions ever become important enough to do that), and 2) run their extension tests and the standard tests against a standard regression test installation and provide evidence that there are no problems before their extensions are included in the mediawiki extensions matrix.
Dan
Hell, we barely have unit tests for Mediawiki itself, much less the many many extensions in SVN. I can't think of a single one, offhand.
FWIW, handling updates between versions is a mess. There are two accepted and documented ways to apply an extension's schema updates. There needs to be one, period. There also needs to be a cleaner Update interface so things like this can be handled more cleanly.
It's nice and great to talk about automated regression testing of the software, but in reality there is no clean way to do it right now. I really admire Gerard and Kim's work on this, but it's really a hack on top of a system that should support this stuff natively.
Regression testing should be automatic, the test cases should be standardized, and extensions should have an easy way to add their own tests to the core set of them. None of these are currently the case. There's a bug open about running parserTests and/or test cases in CodeReview so we can easily and verifiably track regressions in the software. Can't do that until the system makes some sense to begin with :)
-Chad
wikitech-l@lists.wikimedia.org