Hi all,
while MediaWiki has been and is developed primarily with Wikimedia Foundation's interests in mind, there are some big third-party users of MediaWiki out there; while Wikia and wikiHow are the biggest and most well-known, they certainly aren't the only ones.
What's common to third-party users of MediaWiki is not just custom extensions, but sadly core changes, or as they're better known, core hacks -- unsupported changes to the core of the MediaWiki software. I think that everyone will agree with me when I say that we will want to reduce the amount of core hacking by third-parties and instead increase collaboration with us, the upstream developers of MediaWiki.
Reducing the amount of core hacks is generally a good idea for third parties, because it will allow them to upgrade to the latest stable version of MediaWiki easily and things like new hooks can and in many cases are useful to other users of MediaWiki. For example, the MakeGlobalVariablesScript hook ( http://www.mediawiki.org/wiki/Manual:Hooks/MakeGlobalVariablesScript) was originally introduced by Wikia (under the name 'ExtendJSGlobalVars'); in r38397 I added the hook into core under its current name and right now there are many extensions using the hook, including ones used by Wikimedia Foundation sites (see http://www.mediawiki.org/wiki/Category:MakeGlobalVariablesScript_extensions). This is a fine example of how a third-party core hack became a part of the MediaWiki core and thus something useful to other users of MediaWiki, including the Wikimedia Foundation.
Another factor to take into account is security. According to the Version lifecycle page on MediaWiki.org (see http://www.mediawiki.org/wiki/Version_lifecycle), "The release manager has also issued a strong recommendation that versions not listed above as current version or legacy version should not be used in a productive environment. They may contain critical security vulnerabilities and other major bugs, including the threat of possible data loss and/or corruption". For example, wikiHow is running MediaWiki 1.12.0, which was released on 21 March 2008 -- over three years ago. While I'm sure that the wikiHow developers have applied plenty of the more modern security patches, there's still a possibility that they may have missed one patch -- and even if not, MediaWiki 1.12.0 doesn't have all the cool new features that MediaWiki 1.17.0 has. :-)
Essentially I'd like to see all major third-party users contributing code to the upstream version of MediaWiki and everyone keeping their copies of MediaWiki on the official MediaWiki Subversion repository at svn.wikimedia.org. Maybe we could have a branch for each third party under /mediawiki/branches/ or if that's unacceptable, then maybe even a whole new repository (like how we currently have mediawiki, mysql, pywikipedia and wikimedia -- see http://svn.wikimedia.org/viewvc), although I must admit that it sounds a bit overkill to me.
I know from experience that many third parties have written some awesome code and that there are many other people interested in third-party code, but usually getting third-party code to run requires plenty of knowledge about PHP and MediaWiki as the extensions and core changes have usually been designed to work with one site or one farm. I want to change that and bring more extensions available to the general public -- after all, there are many people out there who run a MediaWiki wiki yet they aren't very PHP-savvy. The official MediaWiki Subversion repository is also well-known and it can also act as a "backup" of some kind. I'm sure that most people and companies have extensive backup systems in place, but everything is still possible. For example, the social wiki/blog hybrid site ArmchairGM, where the SocialProfile extension ( http://www.mediawiki.org/wiki/Extension:SocialProfile) and many other, equally cool and interesting extensions were developed, had its own codebase. While the main Wikia codebase has been open source for years, the ArmchairGM codebase was only recently (1 August 2011) open-sourced with the kind help of Sean Colombo -- and for a rather long while, it seemed that ArmchairGM's unique skin and the unique extensions had been lost; now that would've been a major loss for the open source community. Tens of thousands of lines of code, dozens of unique features and some pretty skins were nearly lost; I think that it's in everyone's best interest to prevent such incidents from happening and that is possible by keeping the code free and open.
I've CC'd this message to Sean Colombo of Wikia, Jack Herrick and Reuben Smith of wikiHow, Joachim Bode of Twoonix Software GmbH and Markus Glaser of Hallo Welt! -- please let me know your thoughts about this idea and how your company would be able to contribute.
Thanks and regards, -- Jack Phoenix MediaWiki developer
On Thu, Aug 18, 2011 at 4:07 AM, Jack Phoenix jack@countervandalism.net wrote:
Essentially I'd like to see all major third-party users contributing code to the upstream version of MediaWiki and everyone keeping their copies of MediaWiki on the official MediaWiki Subversion repository at svn.wikimedia.org. Maybe we could have a branch for each third party under /mediawiki/branches/ or if that's unacceptable, then maybe even a whole new repository (like how we currently have mediawiki, mysql, pywikipedia and wikimedia -- see http://svn.wikimedia.org/viewvc), although I must admit that it sounds a bit overkill to me.
I don't think separate branches would even be needed after all that long, for example at the start just so they get a feel for our SVN and CodeReview methods and to work out what their differences are (and progressively interworking their work into the main trunk) but after a while they would most likely be committing their patches straight to core if they are "sane" (as in usable to a variety of people) so it would just be little local "live" hacks that would be different (if they are even needed at all).
Hello,
at Hallo Welt!, we are currently discussing some similar issues. The question is how deep we should and could tie in with mediawiki svn in general. Here are some things that came to my mind:
Changes to core: While BlueSpice is essentially a MediaWiki extension and has a policy of accepting MW core as it is, we do have a handful core hacks where necessary. I doubt, though, that these will make it into core. For example, we use read protection of pages. A list of changes to core code that are necessary in this respect can already be found on mediawiki.org, however, they are not permanently implemented, I assume because of performance questions. Similar issues arise when you want to use real names instead of usernames. Here we'd have to hook at places that seem to be rather performance critical. Having said this, I admit, I never tried to pace those hooks in core. You might call that preemptive policy compliance... or just laziness ;). So the question is, can there be 'flavors' of MediaWiki that do not have to consider the requirements of a high performance high traffic environment? I assume that flavoring MW would get rather complicated. But we could introduce hooks that are only executed if, say, $wgExecuteExpensiveHooks is set.
Another question is about extensions: I would very much like to see BlueSpice in mediawiki svn. There are some difficulties, though. One of them is that we use large third party frameworks and I am not sure I want to maintain them in mediawiki svn (e.g. TinyMCE or ExtJs), nor if I am allowed to do so legally. These could be left out, but then the extension would not be in a working condition when downloaded. Another issue is that we want to maintain release versions of the extension and backport some bug- and security fixes. That basically means branching. And afaik we cannot do that in mediawiki svn. I faintly remember having heard about an architecture, where you can kind of hook a remote repository into some subversion repository. That might be a solution here, since we could keep our own repo and branches while making sure there is an upstream of our trunk version. Actually, this sounds very much like DVCS, but I admit I don't know enough about git and others to know if that would solve this problem. I assume, though, that e.g. Semantic MediaWiki has some similar versioning issues, so I would be very interested to hear their thoughts on that.
Cheers, Markus (mglaser)
Von: Jack Phoenix [mailto:jack@countervandalism.net] Gesendet: Mittwoch, 17. August 2011 20:07 An: Wikimedia developers; Sean Colombo; Jack Herrick; reuben@wikihow.com; joachim.bode@twoonix.com; Markus Glaser Betreff: On third-party users of MediaWiki and collaboration with upstream
Hi all,
while MediaWiki has been and is developed primarily with Wikimedia Foundation's interests in mind, there are some big third-party users of MediaWiki out there; while Wikia and wikiHow are the biggest and most well-known, they certainly aren't the only ones.
What's common to third-party users of MediaWiki is not just custom extensions, but sadly core changes, or as they're better known, core hacks -- unsupported changes to the core of the MediaWiki software. I think that everyone will agree with me when I say that we will want to reduce the amount of core hacking by third-parties and instead increase collaboration with us, the upstream developers of MediaWiki.
Reducing the amount of core hacks is generally a good idea for third parties, because it will allow them to upgrade to the latest stable version of MediaWiki easily and things like new hooks can and in many cases are useful to other users of MediaWiki. For example, the MakeGlobalVariablesScript hook (http://www.mediawiki.org/wiki/Manual:Hooks/MakeGlobalVariablesScript) was originally introduced by Wikia (under the name 'ExtendJSGlobalVars'); in r38397 I added the hook into core under its current name and right now there are many extensions using the hook, including ones used by Wikimedia Foundation sites (see http://www.mediawiki.org/wiki/Category:MakeGlobalVariablesScript_extensions). This is a fine example of how a third-party core hack became a part of the MediaWiki core and thus something useful to other users of MediaWiki, including the Wikimedia Foundation.
Another factor to take into account is security. According to the Version lifecycle page on MediaWiki.org (see http://www.mediawiki.org/wiki/Version_lifecycle), "The release manager has also issued a strong recommendation that versions not listed above as current version or legacy version should not be used in a productive environment. They may contain critical security vulnerabilities and other major bugs, including the threat of possible data loss and/or corruption". For example, wikiHow is running MediaWiki 1.12.0, which was released on 21 March 2008 -- over three years ago. While I'm sure that the wikiHow developers have applied plenty of the more modern security patches, there's still a possibility that they may have missed one patch -- and even if not, MediaWiki 1.12.0 doesn't have all the cool new features that MediaWiki 1.17.0 has. :-)
Essentially I'd like to see all major third-party users contributing code to the upstream version of MediaWiki and everyone keeping their copies of MediaWiki on the official MediaWiki Subversion repository at svn.wikimedia.orghttp://svn.wikimedia.org. Maybe we could have a branch for each third party under /mediawiki/branches/ or if that's unacceptable, then maybe even a whole new repository (like how we currently have mediawiki, mysql, pywikipedia and wikimedia -- see http://svn.wikimedia.org/viewvc), although I must admit that it sounds a bit overkill to me.
I know from experience that many third parties have written some awesome code and that there are many other people interested in third-party code, but usually getting third-party code to run requires plenty of knowledge about PHP and MediaWiki as the extensions and core changes have usually been designed to work with one site or one farm. I want to change that and bring more extensions available to the general public -- after all, there are many people out there who run a MediaWiki wiki yet they aren't very PHP-savvy. The official MediaWiki Subversion repository is also well-known and it can also act as a "backup" of some kind. I'm sure that most people and companies have extensive backup systems in place, but everything is still possible. For example, the social wiki/blog hybrid site ArmchairGM, where the SocialProfile extension (http://www.mediawiki.org/wiki/Extension:SocialProfile) and many other, equally cool and interesting extensions were developed, had its own codebase. While the main Wikia codebase has been open source for years, the ArmchairGM codebase was only recently (1 August 2011) open-sourced with the kind help of Sean Colombo -- and for a rather long while, it seemed that ArmchairGM's unique skin and the unique extensions had been lost; now that would've been a major loss for the open source community. Tens of thousands of lines of code, dozens of unique features and some pretty skins were nearly lost; I think that it's in everyone's best interest to prevent such incidents from happening and that is possible by keeping the code free and open.
I've CC'd this message to Sean Colombo of Wikia, Jack Herrick and Reuben Smith of wikiHow, Joachim Bode of Twoonix Software GmbH and Markus Glaser of Hallo Welt! -- please let me know your thoughts about this idea and how your company would be able to contribute.
Thanks and regards, -- Jack Phoenix MediaWiki developer
On Sun, Aug 21, 2011 at 8:26 AM, Markus Glaser glaser@hallowelt.biz wrote:
Similar issues arise when you want to use real names instead of usernames.
We do have the option in most places to use a real name just its not enabled in core, AFAIK without testing its used in most places where the username is used already, If you find anywhere where it can be useful, file a bug!
One of them is that we use large third party frameworks and I am not sure I want to maintain them in mediawiki svn (e.g. TinyMCE or ExtJs), nor if I am allowed to do so legally. These could be left out, but then the extension would not be in a working condition when downloaded.
If they are already in a SVN repo somewhere we could do a svn external to link it. (We do this for SyntaxHighlight Geshi IIRC).
Another issue is that we want to maintain release versions of the extension and backport some bug- and security fixes. That basically means branching. And afaik we cannot do that in mediawiki svn.
Are you talking about the upstream packages, the MW release or your extension for this? You can do it if you want, we have TAGS to do extensions based on versions and such (the Semantic* extensions so this).
-Peachey
On Wed, Aug 17, 2011 at 2:07 PM, Jack Phoenix jack@countervandalism.net wrote: ....
What's common to third-party users of MediaWiki is not just custom extensions, but sadly core changes, or as they're better known, core hacks -- unsupported changes to the core of the MediaWiki software. I think that everyone will agree with me when I say that we will want to reduce the amount of core hacking by third-parties and instead increase collaboration with us, the upstream developers of MediaWiki.
....
Essentially I'd like to see all major third-party users contributing code to the upstream version of MediaWiki and everyone keeping their copies of MediaWiki on the official MediaWiki Subversion repository at svn.wikimedia.org....
....
Jack Phoenix MediaWiki developer
I've been reading this discussion with interest and thought a few updates/links might be useful:
Sam Reed has put together a list of the Wikia diffs that need to be upstreamed[0], and Sean Colombo's working on it.
Markus Glaser is coming to the upcoming New Orleans hackathon[1] and will chat about third party development upstream to MediaWiki SVN, among other issues.
Olivier Finlay Beaton put together a survey of what versions of MediaWiki are being run out in the wild[2] and found that the majority are on 1.16.x. Some administrators stay on old versions because they have extension compatibility issues, or because they don't know or don't care that new versions are available. We should address those obstacles in other ways. But to the extent that administrators don't upgrade because of their core hacks -- that's the friction that Sean, Jack, Reuben, Joachim, and Markus can especially help us reduce.
From outside our immediate community, an essay on the cost of *not*
upstreaming[3].
[0] http://www.mediawiki.org/wiki/Wikia_code [1] http://mediawiki.org/wiki/NOLA_Hackathon [2] http://www.mediawiki.org/wiki/Manual:Extension_support#What.27s_out_there [3] http://blogs.gnome.org/bolsh/2011/09/01/the-cost-of-going-it-alone/
Sumana Harihareswara Volunteer Development Coordinator Wikimedia Foundation
wikitech-l@lists.wikimedia.org