Un-commingling Libraries in MediaWiki

List overview All Threads
Download

newer

older

m.wikipedia.org - need for admin...

Community vs. centralized...

Trevor Parscal

26 Aug 2010 26 Aug '10

5:57 p.m.

I have been working on the ResourceLoader branch, where I've ended up writing a CSSMin class which performs CSS minification, URI-remapping and data-URI in-lining. It got me thinking that this class would be pretty useful to non-MediaWiki projects too, but sadly we don't have a history of sharing in this way... * Software we've ported to PHP ourselves like our native-PHP CDB implementation or CSSJanus are buried in our code-base, and make use of a couple of trivial wf* global functions, making it somewhat inaccessible to third-party users. Which sucks because third-party users are important! They use the code in their own systems, make improvements and potentially pass them back to us, however if we don't make these things more general-purpose the code will more likely get taken from our repository, tweaked and never passed back; if we don't make it more easily accessible the code will never be found and we won't be taking advantage of the entire PHP development community. Sadness... * Software we've borrowed from other projects like JSMin are also buried within our MediaWiki-proprietary code, and while these libraries can operate independently of MediaWiki, we need to make it clear that they should be kept in sync with their original sources both, upstream and down. * Software we've created is often potentially useful to other projects, but unfortunately tied to and buried within MediaWiki. In some of these cases, the ties to MediaWiki are trivial and could be either optional or removed entirely, and the component could be factored out to a more general-purpose library, available for re-use. I don't have a very mature proposal for solving this completely, but as a first step, it seems like we should have a libraries folder which we can move things that can function in a stand-alone manner to. Initial candidates appear to be libraries that already function in a stand-alone way such as JSMin.php, CSSJanus.php, and CSSMin.php (in the resourceloader branch right now but will be coming to trunk soon). Additional software could be moved into this space after some un-tethering such as Cdb/Cdb_PHP, DjVuImage, etc. Overall, I think it would be great if we could take a look at this and other ways to better share our work with non-MediaWiki projects, and give back to the open-source community. I welcome your thoughts and input. - Trevor

Show replies by date

Aryeh Gregor

26 Aug 26 Aug

6:50 p.m.

On Thu, Aug 26, 2010 at 1:57 PM, Trevor Parscal <tparscal(a)wikimedia.org> wrote:

...

There are already some files that are explicitly meant to be reusable, like IEContentAnalyzer.php and includes/normal/ (both written by Tim Starling). I think this is a nice idea, although I don't think anyone has taken us up on reusing either of those (maybe I'm wrong). For convenience, I guess we want to keep them under phase3/, but maybe putting them in a special folder like phase3/libraries/ would be a good idea. We could try advertising them a bit more, too.

Trevor Parscal

6:53 p.m.

Right - to be clear, I meant phase3/libraries, not moving things completely out of phase3. - Trevor On 8/26/10 11:50 AM, Aryeh Gregor wrote:

...

On Thu, Aug 26, 2010 at 1:57 PM, Trevor Parscal<tparscal(a)wikimedia.org> wrote:

Aryeh Gregor

7:45 p.m.

On Thu, Aug 26, 2010 at 2:53 PM, Trevor Parscal <tparscal(a)wikimedia.org> wrote:

...

Right - to be clear, I meant phase3/libraries, not moving things completely out of phase3.

Yeah -- moving them out of phase3/ would be attractive, so that they really look like separate projects, but probably not worth it.

Platonides

10:58 p.m.

Trevor Parscal wrote:

...

* Software we've ported to PHP ourselves like our native-PHP CDB implementation or CSSJanus are buried in our code-base, and make use of a couple of trivial wf* global functions, making it somewhat inaccessible to third-party users.

Trevor Parscal

11:13 p.m.

You can also use things like function_exists() to conditionally call MediaWiki versions of things or native PHP stuff. - Trevor On 8/26/10 3:58 PM, Platonides wrote:

...

Trevor Parscal wrote:

We should probably provide stub a stub version of those functions, so reusers don't have to declare a bunch of functions just because we call wfProfileIn, wfGetIP and set a couple of hooks. Other problem would be our usage of wrappers like WebRequest. _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Conrad Irwin

11:21 p.m.

On 26 August 2010 16:13, Trevor Parscal <tparscal(a)wikimedia.org> wrote:

...

You can also use things like function_exists() to conditionally call MediaWiki versions of things or native PHP stuff.

With respect, this is the wrong way to go about it. Unconditionally remove all MediaWIki from within the library, and, if necessary, create a wrapper that provides the extra functionality we need. Obviously, if it'd need a huge wrapper compared to the size of the library, it's not worth extracting - the chances are that any re-users will need to create vast wrappers too. The current examples of deliberately MediaWIki free code are very good candidates for this as they provide complex functionality that's really hard to do right yet they have very simple interfaces. Conrad

...

- Trevor On 8/26/10 3:58 PM, Platonides wrote:

Trevor Parscal wrote:

_______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Chad

11:25 p.m.

On Thu, Aug 26, 2010 at 7:21 PM, Conrad Irwin <conrad.irwin(a)gmail.com> wrote:

...

On 26 August 2010 16:13, Trevor Parscal <tparscal(a)wikimedia.org> wrote:

You can also use things like function_exists() to conditionally call MediaWiki versions of things or native PHP stuff.

With respect, this is the wrong way to go about it. Unconditionally remove all MediaWIki from within the library, and, if necessary, create a wrapper that provides the extra functionality we need.

This.

...

The current examples of deliberately MediaWIki free code are very good candidates for this as they provide complex functionality that's really hard to do right yet they have very simple interfaces.

IEContentAnaylzer is probably the best example. CDB-related stuff too. I went ahead and moved some stuff around in r71763. -Chad

Platonides

27 Aug 27 Aug

12:54 p.m.

Chad wrote:

...

IEContentAnaylzer is probably the best example. CDB-related stuff too. I went ahead and moved some stuff around in r71763. -Chad

What about relicensing it at the same time? To BSD or so. I think IEContentAnaylzer is already OSI-compatible (the pieces added on r44505 from had been added in 44505) so it would be just changing the header notice. I don't think r63254 qualifies for copyright (case change for doxygen), but for extra correctness the ok by ialex would be nice.

Aryeh Gregor

5:42 p.m.

On Fri, Aug 27, 2010 at 8:54 AM, Platonides <Platonides(a)gmail.com> wrote:

...

What about relicensing it at the same time? To BSD or so.

Why?

Platonides

8:31 p.m.

Aryeh Gregor wrote:

...

On Fri, Aug 27, 2010 at 8:54 AM, Platonides <Platonides(a)gmail.com> wrote:

What about relicensing it at the same time? To BSD or so.

Why?

a) To encourage reuse and b) Avoid debates about using the libraries with other non-GPL code (eg. under PHP license) License of each file in libs would depend on each the module creator, of course.

Aryeh Gregor

9:12 p.m.

On Fri, Aug 27, 2010 at 4:31 PM, Platonides <Platonides(a)gmail.com> wrote:

...

a) To encourage reuse and b) Avoid debates about using the libraries with other non-GPL code (eg. under PHP license)

The exact same considerations apply to MediaWiki as a whole, though, and there are also considerations in favor of GPL. Clearly some people prefer BSD, but what does this have to do with breaking code out into libraries? It seems orthogonal, so I was puzzled by you mentioning it in this particular context. (You can license your files as BSD if you like even if they're in MediaWiki core, or at least some people have done that.)

Trevor Parscal

9 Sep 9 Sep

8:08 p.m.

On 8/26/10 4:25 PM, Chad wrote:

...

On Thu, Aug 26, 2010 at 7:21 PM, Conrad Irwin<conrad.irwin(a)gmail.com> wrote:

On 26 August 2010 16:13, Trevor Parscal<tparscal(a)wikimedia.org> wrote:

You can also use things like function_exists() to conditionally call MediaWiki versions of things or native PHP stuff.

With respect, this is the wrong way to go about it. Unconditionally remove all MediaWIki from within the library, and, if necessary, create a wrapper that provides the extra functionality we need.

This.

The current examples of deliberately MediaWIki free code are very good candidates for this as they provide complex functionality that's really hard to do right yet they have very simple interfaces.

IEContentAnaylzer is probably the best example. CDB-related stuff too. I went ahead and moved some stuff around in r71763. -Chad _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Following your (chad's) lead, I've now moved some more stuff to includes/libs -- r72678 * JSMin -- Ryan Grove's PHP port (http://github.com/rgrove/jsmin-php/) of Douglas Crockford's JSMin library (http://www.crockford.com/javascript/jsmin.html) for removing white-space and comments from JavaScript * CSSJanus -- Roan Kattouw's PHP port of Google's CSSJanus library (http://code.google.com/p/cssjanus/) for transforming left-to-right style-sheets into right-to-left orientation * CSSMin -- Trevor Parscal's (my) library for CSS minification, URL remapping and data-URL embedding There are more Gems in MediaWiki which can likely be moved here too, but this is at least a good start. - Trevor

Dmitriy Sintsov

27 Aug 27 Aug

7:23 a.m.

* Platonides <Platonides(a)gmail.com> [Fri, 27 Aug 2010 00:58:25 +0200]:

...

Trevor Parscal wrote: > * Software we've ported to PHP ourselves like our native-PHP CDB > implementation or CSSJanus are buried in our code-base, and

make

...

use of a couple of trivial wf* global functions, making it somewhat inaccessible to third-party users.

We should probably provide stub a stub version of those functions, so reusers don't have to declare a bunch of functions just because we

call

...

wfProfileIn, wfGetIP and set a couple of hooks. Other problem would be our usage of wrappers like WebRequest.

I like WebRequest. If there was an request router, in similar way to CakePHP and another frameworks, it would even be better. Sorry if that's a bit off-topic. Dmitriy

Tim Starling

1:46 a.m.

On 27/08/10 03:57, Trevor Parscal wrote:

...

You should put it in PEAR. Very few people are going to find it if it's just in the MediaWiki core, regardless of what directory you put it in. -- Tim Starling

Tim Starling

2:41 a.m.

I wrote:

...

You should put it in PEAR. Very few people are going to find it if it's just in the MediaWiki core, regardless of what directory you put it in.

I wouldn't like to be accused of hypocrisy, so I released a new version of File_Ogg: http://pear.php.net/package/File_Ogg It's not too hard. -- Tim Starling

Trevor Parscal

3:40 p.m.

So, you suggest we maintain it in 2 places (where we use it in MediaWiki core and where people have access to it), and ask that others push changes back to where? This only solves part of the problem, unless we stop including these libraries in core and make MediaWiki dependent on some PEAR modules, which seems undesirable. - Trevor On 8/26/10 6:46 PM, Tim Starling wrote:

...

On 27/08/10 03:57, Trevor Parscal wrote:

You should put it in PEAR. Very few people are going to find it if it's just in the MediaWiki core, regardless of what directory you put it in. -- Tim Starling _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Rob Lanphier

4:44 p.m.

...

On 8/26/10 6:46 PM, Tim Starling wrote:

On 27/08/10 03:57, Trevor Parscal wrote: > It got me thinking that [the CSSmin class] would be > pretty useful to non-MediaWiki projects too, but sadly we don't have a > history of sharing in this way...

...

You should put it in PEAR. Very few people are going to find it if it's just in the MediaWiki core, regardless of what directory you put it in.

...

So, you suggest we maintain it in 2 places (where we use it in MediaWiki core and where people have access to it), and ask that others push changes back to where?

Tim's right...putting it in PEAR is a great way to signal "this is a reusable component; please reuse this". PEAR isn't a source of truth; it's a distribution channel. In order to get discovered and used, libraries need to be in a distribution channel that people use. Many people use PEAR, other people use Linux distribution repositories. svn.wikimedia.org would continue to be the original source repository for the library, but PEAR would be one of many distribution channels. The PEAR version would only get updated each time you release the library. Rob

Trevor Parscal

5:25 p.m.

My understanding is that PEAR has a specific coding style and set of standards, which is different than that of MediaWiki. Also things like JSMin should not be hijacked and thrown into PEAR -- it came from http://github.com/rgrove/jsmin-php/ -- it already has a distribution chanel. In other words, the idea of using PEAR works really well for things we took from PEAR or think we should specifically contribute to PEAR, but not for all cases that I originally presented. Moving stand-alone things into a specific place, such as what ^demon has done in r 71774, is still useful as at least a first step towards sharing and generally being better about giving back to the larger open source community. I do however agree with Tim however that simply moving libraries to phase3/includes/libs, while introducing a more sanitary convention for 3rd party library inclusion, does not by itself raise awareness to the availability of the libraries - that part of the problem may need to be solved in multiple ways, only one of which is likely to be PEAR. - Trevor On 8/27/10 9:44 AM, Rob Lanphier wrote:

...

On 8/26/10 6:46 PM, Tim Starling wrote: > On 27/08/10 03:57, Trevor Parscal wrote: >> It got me thinking that [the CSSmin class] would be >> pretty useful to non-MediaWiki projects too, but sadly we don't have a >> history of sharing in this way...

> You should put it in PEAR. Very few people are going to find it if > it's just in the MediaWiki core, regardless of what directory you put > it in.

So, you suggest we maintain it in 2 places (where we use it in MediaWiki core and where people have access to it), and ask that others push changes back to where?

mah＠everybody.org

28 Aug 28 Aug

3:45 p.m.

Trevor Parscal <tparscal(a)wikimedia.org> writes:

...

My understanding is that PEAR has a specific coding style and set of standards, which is different than that of MediaWiki.

Coding style is flexible, especially if we're the ones doing the distribution. By this, I mean that “putting it in PEAR” could just mean setting up a MediaWiki PEAR channel. PHPUnit, for example, moved http://pear.php.net/package/PHPUnit to its own PEAR channel http://pear.php.net/package/PHPUnit. Other packages related to PHPUnit are also maintained there. And as Rob pointed out, putting these “sharable artifacts” in a PEAR channel doesn't mean that we can't continue to maintain them in SVN. Setting up pear.mediawiki.org seems like a good solution to me. Mark. -- http://hexmode.com/ Embrace Ignorance. Just don't get too attached.

Platonides

6:17 p.m.

Mark A. Hershberger wrote:

...

Trevor Parscal <tparscal(a)wikimedia.org> writes:

My understanding is that PEAR has a specific coding style and set of standards, which is different than that of MediaWiki.

I think the second should be http://pear.phpunit.de/

...

And as Rob pointed out, putting these “sharable artifacts” in a PEAR channel doesn't mean that we can't continue to maintain them in SVN. Setting up pear.mediawiki.org seems like a good solution to me. Mark.

That's an interesting idea. But I'm not sure if it would achieve the "make it discoverable" goal.

Trevor Parscal

29 Aug 29 Aug

1:12 a.m.

On 8/28/10 11:17 AM, Platonides wrote:

...

Mark A. Hershberger wrote:

Trevor Parscal<tparscal(a)wikimedia.org> writes:

My understanding is that PEAR has a specific coding style and set of standards, which is different than that of MediaWiki.

I think the second should be http://pear.phpunit.de/

That's an interesting idea. But I'm not sure if it would achieve the "make it discoverable" goal. _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

It does make it accessible though! - Trevor

mah＠everybody.org

30 Aug 30 Aug

2:40 p.m.

Platonides <Platonides(a)gmail.com> writes:

...

That's an interesting idea. But I'm not sure if it would achieve the "make it discoverable" goal.

“Make it discoverable” is more of a social problem, though. It can be solved by using the visibility of blogs, etc. Talking about the libraries and using them in other projects will help. Mark. -- http://hexmode.com/ Embrace Ignorance. Just don't get too attached.

Aryeh Gregor

4:11 p.m.

On Mon, Aug 30, 2010 at 10:40 AM, Mark A. Hershberger <mah(a)everybody.org> wrote:

...

“Make it discoverable” is more of a social problem, though. It can be solved by using the visibility of blogs, etc. Talking about the libraries and using them in other projects will help.

Getting them in standard searchable, easy-to-use distribution channels like pear.php.net will also help a lot. I think it's fine if we reformat them to follow PEAR standards instead of MediaWiki standards for that purpose.

4978

days inactive

4992

days old

wikitech-l@lists.wikimedia.org

Manage subscription

23 comments

9 participants

tags (0)

participants (9)

Aryeh Gregor
Chad
Conrad Irwin
Dmitriy Sintsov
mah＠everybody.org
Platonides
Rob Lanphier
Tim Starling
Trevor Parscal