Hi all, I am wondering if there is any interest in some mods to enhance MediaWikis ability to be multi-hosted from a single core of code ?
I've done this a few times now with phpBB2 and Wordpress and mostly have the "pattern" down on how to do it. I just spent an hour with MediaWiki and there are not too many changes involved. Is there any interest in this kind of thing and if so where is the best place to post the details... wiki, here ?
--markc
On Nov 13, 2004, at 8:21 PM, Mark Constable wrote:
Hi all, I am wondering if there is any interest in some mods to enhance MediaWikis ability to be multi-hosted from a single core of code ?
There's some documentation on how we have things set up at Wikipedia here: http://wp.wikidev.net/Wiki_farm
-- brion vibber (brion @ pobox.com)
On Sunday 14 November 2004 14:24, Brion Vibber wrote:
On Nov 13, 2004, at 8:21 PM, Mark Constable wrote:
Hi all, I am wondering if there is any interest in some mods to enhance MediaWikis ability to be multi-hosted from a single core of code ?
There's some documentation on how we have things set up at Wikipedia here: http://wp.wikidev.net/Wiki_farm
Thanks for that, a good read but a bit obscure looking at it from the outside. My observation so far is that it would not take that much change to the distributed code to allow the installer to offer a multi-hosting setup that anyone could take advantage of... in our case that will be 300 vhosts and 4000 subdomains. Do you, or anyone, think there is any interest in getting multi-hosting mods into the official mainstream code ?
--markc
Mark Constable wrote:
On Sunday 14 November 2004 14:24, Brion Vibber wrote:
On Nov 13, 2004, at 8:21 PM, Mark Constable wrote:
Hi all, I am wondering if there is any interest in some mods to enhance MediaWikis ability to be multi-hosted from a single core of code ?
There's some documentation on how we have things set up at Wikipedia here: http://wp.wikidev.net/Wiki_farm
Thanks for that, a good read but a bit obscure looking at it from the outside. My observation so far is that it would not take that much change to the distributed code to allow the installer to offer a multi-hosting setup that anyone could take advantage of... in our case that will be 300 vhosts and 4000 subdomains. Do you, or anyone, think there is any interest in getting multi-hosting mods into the official mainstream code ?
Perhaps you would like to tell us what you actually intend to do? I understand it might be obscure from the outside, but our apache configuration files have been publically available for some time. I just uploaded the remainder of our wiki configuration, and made an index file:
I can't see any problems scaling up our configuration from 500 to 4000 subdomains. What problems have you encountered?
-- Tim Starling
On Monday 15 November 2004 11:16, Tim Starling wrote:
Perhaps you would like to tell us what you actually intend to do?
The ideal, for me, would be that a single SIMPLE index.php dropped into any on-web doc root pulls up a unique full working instance of MediaWiki. That 99.9% of the code works properly from an off-web includes area and, but not so importantly, that on-web CSS and image requirements can come from some default initial domain so that nothing other than the index.php and a LocalSettings.php are required in any vhost webfolder...
<?php require './LocalSettings.php'; require 'mediawiki.php'; ?>
I understand it might be obscure from the outside, but our apache configuration files have been publically available for some time. I just uploaded the remainder of our wiki configuration, and made an index file:
Good one, that backs up the textual explanation nicely.
I can't see any problems scaling up our configuration from 500 to 4000 subdomains. What problems have you encountered?
None yet. I can easily hack the code to make it work for our particular system setup close to my ideal spec above, but, that makes future upgrades a risky process. The current install basically assumes that all the required parts are within the same directory as the index.php. The simplistic approach to creating a second installation, on the same server, would be to repeat the current installation procedure and create a separate completely independant installation in another webfolder. That's fine up to about a dozen installs and then it gets out of hand in both the installation procedure and any future upgrades.
With my first attempt to make MediaWiki work, like my specified "ideal" above, it seemed to me that it _MAY_ not take much effort to mildly alter the front end controller and installation procedure to take into account either a simple single installation OR a more complex multi-hosting installation. If so, and it became an official core part of the MediaWiki project, then there would be no reason not to use MediaWiki in larger multi-hosted environments. It can be now with some tinkering but I can see that even that tinkering could be absorbed into an official installation procedure.
My first attempt was to create a default installation then move everything inside an offweb include path and start with a simple <?php include 'mediawiki.php' ?> (being the current index.php) and put back into the current webfolder what was essential to bring up any page at all. The simplest minimum, without worrying about a default webhost for images and stylesheets, is...
LocalSettings.php images/ index.php redirect.php@ (a link -> index.php) stylesheets/ tmp/ (was autocreated)
I changed $IP in LocalSettings.php to the off-web include dir and commented out the initial part of mediawiki.php (original index.php) to just below the "require LocalSettings.php" and now my current index.php used above is...
<?php error_log(var_export($_REQUEST, true));
$wgRequestTime = microtime(); unset( $IP ); @ini_set( "allow_url_fopen", 0 ); # For security... if(!file_exists("LocalSettings.php")) { die( "You'll have to <a href='config/index.php'>set the wiki up</a> first!" ); } define( "MEDIAWIKI", true ); require_once( "./LocalSettings.php" );
if (isset($_GET['redirect']) and $_GET['redirect'] == 'Go') { # unset( $DP ); # unset( $IP ); $wgCommandLineMode = false; # require_once( "./LocalSettings.php" ); global $wgArticlePath; require_once( "WebRequest.php" ); $wgRequest = new WebRequest(); $page = $wgRequest->getVal( "wpDropdown" ); $url = str_replace( "$1", urlencode( $page ), $wgArticlePath ); header( "Location: {$url}" ); } else { include 'mediawiki.php'; } ?>
This is my first cut at it in the first hour of installing MediaWiki to see what might be involved in doing this. It's in no way working properly or anywhere near deployed. I was pleased that I did not have to alter too much code to get to this point and it occurred to me that it might be possible to continue this move through "official channels" to get something like this into the core code.
--markc
On Nov 14, 2004, at 6:55 PM, Mark Constable wrote:
The ideal, for me, would be that a single SIMPLE index.php dropped into any on-web doc root pulls up a unique full working instance of MediaWiki. That 99.9% of the code works properly from an off-web includes area and, but not so importantly, that on-web CSS and image requirements can come from some default initial domain so that nothing other than the index.php and a LocalSettings.php are required in any vhost webfolder...
<?php require './LocalSettings.php'; require 'mediawiki.php'; ?>
What's stopping that from working?
The simplest minimum, without worrying about a default webhost for images and stylesheets, is...
LocalSettings.php images/ index.php redirect.php@ (a link -> index.php) stylesheets/ tmp/ (was autocreated)
Neither images/ nor stylesheets/ should be necessary if you don't want them there; you can set the on-disk directories for each arbitrarily and give them arbitrary URL path prefixes from LocalSettings.php. I'm not sure what the tmp directory is for?
index.php, redirect.php, and LocalSettings.php can be symbolic links to a central holding area if you like.
For that matter, since none of these files are domain-specific you don't actually need 4000 directories with 4000 copies for 4000 domains. Your LocalSettings.php (or whatever you load up from there) can set up the appropriate configuration based on the virtual server's hostname or environment variables set from Apache.
-- brion vibber (brion @ pobox.com)
I have been following this MultiHosting thread, as well as the hooks and filters thread with great interest.
I think that one of the most crucial aspects of successfully integrating open source solutions is treating the core distribution as "read only". Mediawiki 1.3x makes it very difficult to adhere to this principle - it is too tempting, (often irresistible ;-)), to modify the core distribution.
Performing in-place customizations makes upgrades very very difficult, and worse, makes people's problem idiosyncratic (ruining most hope for support from the community, or return contributions).
I think one of the primary design goals 1.4 and up should be designing software that does not require, nor encourage, the direct modification of the core distribution.
Pluggable auth, the proposed, the extensions and special file handling, and especially the hooks and filters proposal are steps in the right direction.
When it came to multiple hosting, I had a choice of following the mediawiki setup, or the helpful suggestion of Mark Constable (Thanks!). While the mediawiki setup might ultimately provide ttw instance creation, it also presents its own shortcomings. Mark's suggestion, with a little interpolation, allows for a great deal of flexibility. It allows for the creation of lightweight wiki instances that run off a common software installation. Likewise, its also makes it easier to manage the customizations on the file system using standard version control tools, as well as manage code from across development/staging/production environments.
I am not talking about treating the wiki as a cms or even development platform, but if you want to do extensive customizations of the skins, or involve more than one developer on a wiki project, this kind of control is essential.
Here is the setup that I am currently using:
$SOFTWARE_HOME/<standard untarred mediawiki>
$INSTANCE_HOME/InstanceSettings.php $INSTANCE_HOME/bootstrap.php $INSTANCE_HOME/images/ $INSTANCE_HOME/includes/SkinMyCustomSkin.php $INSTANCE_HOME/index.php $INSTANCE_HOME/redirect.php $INSTANCE_HOME/stylesheets/myCustomSkin $INSTANCE_HOME/templates/
where index.php is: ------------------- <?php error_log(var_export($_REQUEST, true));
$wgRequestTime = microtime(); unset( $IP ); ini_set( "allow_url_fopen", 0 ); # For security... if(!file_exists("./InstanceSettings.php")) { die( "You need to configure your sandbox before this wiki can work" ); }
define( "MEDIAWIKI", true ); require_once("./bootstrap.php"); require_once( "$SOFTWARE_HOME/LocalSettings.php" ); require_once( "./InstanceSettings.php" ); # inlcude our skin, so that we don't have to modify core code require_once("SkinCCNMTL.php");
# mediawiki.php is the same as the original index.php, with the top few lines c ommented out require 'mediawiki.php';
?>
redirect.php is a straight copy of the one in software_home (it needs to be a copy, not a link, or the ./LocalSettings.php include will happen inside the software_home directory, not the instance_home directory.
bootstrap.php is ------------------- <?php # # this file definesthe environemnt specific paths and resources, per sandbox
$IP = $SOFTWARE_HOME = "/usr/local/share/sandboxes/common/mediawiki-src"; $INSTANCE_HOME = "/usr/local/share/sandboxes/jonah/uwpwiki"; $INSTANCE_URL_BASE = "http://wiki.columbia.edu:8080"; $INSTANCE_URL_PATH = "/sb/jonah/mediawiki";
ini_set( "include_path", ".:$INSTANCE_HOME:$INSTANCE_HOME/includes:$INSTANCE_HO ME/templates:$IP:$IP/includes:$IP/languages" );
?>
The only modification I needed to make in the core distro was copying index.php -> mediawiki.php and commenting out the first ~15 lines (up until requiring includes/Setup.php).
This index.php combined with the include_path allows me to set up sandbox specific settings, instance specific settings, as well as server wide, global settings (in the LocalSettings and DefaultSettings).
If the core mediawiki introduced a similar construct, I wouldn't even need to modify the original index.php inside of software home.
I think this kind of structure would encourage people to use more OO techniques such as extension (w/in your own instance's include path, which is picked up before the software home one), and in general, keep folks away from the core distribution.
Thanks for all your help.
Jonah
On Nov 29, 2004, at 10:41 PM, Jonah Bossewitch wrote: [snip]
This index.php combined with the include_path allows me to set up sandbox specific settings, instance specific settings, as well as server wide, global settings (in the LocalSettings and DefaultSettings).
There's no need to change index.php, this is what LocalSettings.php is for. (Nor do we need more than one of it, as site-specific setup logic happens under it and its includes.)
I guess it just seems a bit weird to have people tell us why they can't use our software to run multiple sites off one set of source when running multiple sites off one set of source is exactly what we do day after day.
-- brion vibber (brion @ pobox.com)
There's no need to change index.php, this is what LocalSettings.php is for. (Nor do we need more than one of it, as site-specific setup logic happens under it and its includes.)
Maybe I am confused, but as I understood it, the wikipedia site itself is running with an alternative configuration, described here:
http://wp.wikidev.net/Wiki_farm
and using this additional configuration http://wikimedia.org/conf/
and presumably, a modified index.php or LocalSettings that initializes and uses the SiteConfiguration object. The 1.3.8 download does not do ordinarily use of the SiteConfiguration object.
But, I think I didn't do a good job of explaining the difference between running multiple sites all with the same policy versus running multiple sites with different policies.
As a very typical use case, imagine trying to power two (or more) mediawiki instances on the same server that are branded completely differently. One way to model this is for each of them to have their own skin, which can be worked on separately and independently by webdesign specialists. Creating a custom phptal skin requires extending the SkinPHPTal class. If you are managing custom code for two different installations, they really should be in their own versioned repositories, not intermingled.
What's the alternative? Hacking the Skin.php or SkinPHPTal.php directly in the single, server-wide, software_home?
Beyond that, you may want to extend different mediawiki instances (special pages, extensions, etc) differently for each project (not in a server-wide way). So some special pages may exist on wiki A, but not on wiki B, but you would still like for wiki A and B to be running off the same set of source.
To top it off, once you begin this kind of development, its really nice to be able to test it, stage it, and deploy it in separate environments.
This is the reasoning that led me to set up a separate, minimal footprint and include path on a per-instance basis, ''not'' a per-server basis.
Do other developers building wikis using the mediawiki software typically version their entire wiki codebase themselves? How do they easily manage and track the patches that they apply to the mediawiki, if they don't cleanly separate out their changes?
Perhaps one of the reasons that the wikipedia team doesn't encounter this difficulty in practice, is because whenever the wikipedia needs a new feature it ends up in the core, and furthermore, it is usually applicable to all the sites that the wikipedia powers.
I guess it just seems a bit weird to have people tell us why they can't use our software to run multiple sites off one set of source when running multiple sites off one set of source is exactly what we do day after day.
Is this really such a weird requirement? I am fairly sure there is no easy way to manage the collaborative development of multiple custom wikis without giving each of them their own "private" space. The challenge is doing so w/out having to replicate the core code for each instance.
Thanks for responding. Please let me know if you think I am being thick.
Best Regards, Jonah
On Nov 30, 2004, at 10:50 PM, Jonah Bossewitch wrote:
There's no need to change index.php, this is what LocalSettings.php is for. (Nor do we need more than one of it, as site-specific setup logic happens under it and its includes.)
Maybe I am confused, but as I understood it, the wikipedia site itself is running with an alternative configuration, described here:
http://wp.wikidev.net/Wiki_farm
and using this additional configuration http://wikimedia.org/conf/
Yes, that's a description of how we use MediaWiki with a single copy of the source to run several hundred wikis with varying configurations.
and presumably, a modified index.php
Nope.
or LocalSettings that initializes and uses the SiteConfiguration object.
Yes! That's the purpose of LocalSettings.php; to load all customizations.
The 1.3.8 download does not do ordinarily use of the SiteConfiguration object.
The common case is a single installation on a hosted account where there is limited control and fancy stuff won't work. Thus, the default installation is the simple, reliable in-place system.
For those of us doing big multi-wiki installations, we can relatively easily move the files, set up a global include_path, and whip up a multi-configuration LocalSettings.php file to suit our needs. The SiteConfiguration object is a multi-configuration helper that Tim came up with to make this easier, but it's just one way to do it.
As a very typical use case, imagine trying to power two (or more) mediawiki instances on the same server that are branded completely differently. One way to model this is for each of them to have their own skin, which can be worked on separately and independently by webdesign specialists. Creating a custom phptal skin requires extending the SkinPHPTal class.
You would probably do this with separate subclassed skins for each "brand", rather than hacking the base class.
You wouldn't need even this if customizing the style sheets and messages is sufficient; this can be done purely from within the wiki interface (MediaWiki:Monobook.css, MediaWiki:MySkin.css, etc), or you can hack up a separate stylesheets directory and use the appropriate one for each wiki without changing any of the base source.
If you are managing custom code for two different installations, they really should be in their own versioned repositories, not intermingled.
If you have set yourself the requirement of having separate version-controlled sets of source, then by definition you're going to have some difficulty running them both from a common set of source. I don't think this should be necessary, however.
Beyond that, you may want to extend different mediawiki instances (special pages, extensions, etc) differently for each project (not in a server-wide way). So some special pages may exist on wiki A, but not on wiki B, but you would still like for wiki A and B to be running off the same set of source.
Extensions are to be loaded from LocalSettings.php, so you can load them conditionally.
I guess it just seems a bit weird to have people tell us why they can't use our software to run multiple sites off one set of source when running multiple sites off one set of source is exactly what we do day after day.
Is this really such a weird requirement?
It's not weird at all; it's exactly what we do with it after all. What's weird is when we get told that we can't possibly be doing what we're doing because it's impossible.
I am fairly sure there is no easy way to manage the collaborative development of multiple custom wikis without giving each of them their own "private" space. The challenge is doing so w/out having to replicate the core code for each instance.
I'm not sure what that means, so I can't really comment on it.
-- brion vibber (brion @ pobox.com)
wikitech-l@lists.wikimedia.org