Hey,
Over the past few months a number of people have been poking at Composer [0] support for MediaWiki [1]. Today I had a look at this and found that though we are close to getting this to work, there are a few remaining problems to be tackled.
1. MediaWiki needs to load the composer autoloader when present.
There currently already is some code in core to do this, though this code is placed at a to early point, resulting in things registered by extensions getting overridden by DefaultSettings.
I made an attemtp to fix this [2] by placing the inclusion of the autoloader after the one of LocalSettings (see patchset 2). As Hashar pointed out, this prevents one from changing the default configuration of extensions. Inclusion of the autoloader thus needs to happen before the end of LocalSettings. It also needs to happen after the start of LocalSettings, since some extensions need the core configuration to be set already or require the user to define things before they get included. This basically means the only place where we can put it is in LocalSettings, on the same place where people typically include extensions. This is what is done in PS3. Does anyone see a better way to do this?
2. Installing extensions leaves the composer.json file modified.
When installing one or more extensions via Composer, they will get added to the require section in composer.json. composer.json is not in the gitignore list. So you might well run into conflicts here, and in any case will have a modified file that is tracked in your git repo, which is annoying. In case of extension installation via LocalSettings, this stuff is in a file that is in gitignore. We could just add composer.json there as well, but this means that when we make changes to it on master, people will not get them any time soon. This is problematic in case we where to make MW core dependent on other packages, though this seems unlikely to happen, and is thus perhaps just a theoretical concern. Does anyone see a way around this problem better than putting composer.json in gitignore? Any concerns with putting it in gitignore?
3. Not clear how to best install an extension
The best command to use for installation of an extension when you already have MediaWiki seems to be "composer require", for instance "composer require ask/ask:dev-master". When using this command, apparently the require-dev packages are also installed. Since MW specifies PHPUnit in require-dev, it is installed, together with all of its dependencies (quite some code) for no good reason. In case of the require command, there appears to be no way to specify it should not get the dev packages.
An alternate approach to installation is to add the things to install manually in the require section of composer.json and do a "composer update --no-dev". This approach might be fine when doing a manual install, though it clearly does not work well when you want to automate it (ie in a CI build process).
Again, does anyone know of a better way to do such an install? And if not, perhaps we can simply get rid of the require-dev section in our composer.json file so we do not run into the problem its causing when using "composer require"?
[0] http://getcomposer.org/ [1] https://www.mediawiki.org/wiki/Composer [2] https://gerrit.wikimedia.org/r/#/c/71616/
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On Tue, Jul 2, 2013 at 12:30 PM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
It also needs to happen after the start of LocalSettings, since some extensions need the core configuration to be set already or require the user to define things before they get included.
This is an issue with the extensions. The expectation for extensions is that when the extension file is included MediaWiki is not set up. If an extension needs to do things *after* MW is set up, it should be adding a callback to $wgExtensionFunctions, which is called from the end of Setup.php.
Does anyone see a way around this
problem better than putting composer.json in gitignore? Any concerns with putting it in gitignore?
Not sure on how to solve this one. It's especially troublesome for people who don't use git (i.e., they just extract the tarballs to upgrade, which means composer.json will just be completely overwritten).
The best command to use for installation of an extension when you already
have MediaWiki seems to be "composer require", for instance "composer require ask/ask:dev-master". When using this command, apparently the require-dev packages are also installed.
That's weird. There's no way to install the non-dev of an extension?
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
Hey,
This is an issue with the extensions. The expectation for extensions is
that when the extension file is included MediaWiki is not set up. If an extension needs to do things *after* MW is set up, it should be adding a callback to $wgExtensionFunctions, which is called from the end of Setup.php.
I'm absolutely fine with declaring this to be unsupported. Being an active extension developer myself, it was however not clear to me this was the case. I suspect the same is true for most other extension developers, and that thus, by extension, a bunch of extensions are currently violating this. I can't think of any out of the top of my head though, so perhaps I'm overestimating the number of violations.
If people in general are fine with this, we can move the inclusion of the composer autoloader to right before LocalSettings.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On Tue, 02 Jul 2013 09:30:57 -0700, Jeroen De Dauw jeroendedauw@gmail.com wrote:
- Installing extensions leaves the composer.json file modified.
When installing one or more extensions via Composer, they will get added to the require section in composer.json. composer.json is not in the gitignore list. So you might well run into conflicts here, and in any case will have a modified file that is tracked in your git repo, which is annoying. In case of extension installation via LocalSettings, this stuff is in a file that is in gitignore. We could just add composer.json there as well, but this means that when we make changes to it on master, people will not get them any time soon. This is problematic in case we where to make MW core dependent on other packages, though this seems unlikely to happen, and is thus perhaps just a theoretical concern. Does anyone see a way around this problem better than putting composer.json in gitignore? Any concerns with putting it in gitignore?
This is precisely one of the reasons I thought installing MW extensions via Composer was a stupid idea in the first place. Installing extensions should not require the file that MediaWiki declares it's library dependencies and information about itself to be modified.
Another one is extension assets. Have you dealt with the fact that MW doesn't have a proper path to serve the css/js in debug mode, images to IE, and other assets that an extension needs.
- Not clear how to best install an extension
The best command to use for installation of an extension when you already have MediaWiki seems to be "composer require", for instance "composer require ask/ask:dev-master". When using this command, apparently the require-dev packages are also installed. Since MW specifies PHPUnit in require-dev, it is installed, together with all of its dependencies (quite some code) for no good reason. In case of the require command, there appears to be no way to specify it should not get the dev packages.
An alternate approach to installation is to add the things to install manually in the require section of composer.json and do a "composer update --no-dev". This approach might be fine when doing a manual install, though it clearly does not work well when you want to automate it (ie in a CI build process).
Again, does anyone know of a better way to do such an install? And if not, perhaps we can simply get rid of the require-dev section in our composer.json file so we do not run into the problem its causing when using "composer require"?
No. One of the explicit changes was to support PHPUnit installed via composer instead of globally (which can be quite awkward to do sometimes). And I have a gerrit change in-waiting which has tests that use a library that's optionally installed via `composer install --dev`.
[0] http://getcomposer.org/ [1] https://www.mediawiki.org/wiki/Composer [2] https://gerrit.wikimedia.org/r/#/c/71616/
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 -- _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Tue, Jul 2, 2013 at 4:22 PM, Daniel Friesen daniel@nadir-seen-fire.comwrote:
On Tue, 02 Jul 2013 09:30:57 -0700, Jeroen De Dauw jeroendedauw@gmail.com wrote:
- Installing extensions leaves the composer.json file modified.
When installing one or more extensions via Composer, they will get added to the require section in composer.json. composer.json is not in the gitignore list. So you might well run into conflicts here, and in any case will have a modified file that is tracked in your git repo, which is annoying. In case of extension installation via LocalSettings, this stuff is in a file that is in gitignore. We could just add composer.json there as well, but this means that when we make changes to it on master, people will not get them any time soon. This is problematic in case we where to make MW core dependent on other packages, though this seems unlikely to happen, and is thus perhaps just a theoretical concern. Does anyone see a way around this problem better than putting composer.json in gitignore? Any concerns with putting it in gitignore?
This is precisely one of the reasons I thought installing MW extensions via Composer was a stupid idea in the first place. Installing extensions should not require the file that MediaWiki declares it's library dependencies and information about itself to be modified.
We can't be the only people in the world with this problem. Instead of abandoning an excellent attempt by the PHP community to provide alternatives to NIH can we lend some expertise in solving the issue instead?
Another one is extension assets. Have you dealt with the fact that MW doesn't have a proper path to serve the css/js in debug mode, images to IE, and other assets that an extension needs.
ditto
Erik Bernhardson
On Tue, 02 Jul 2013 17:27:50 -0700, Erik Bernhardson ebernhardson@wikimedia.org wrote:
On Tue, Jul 2, 2013 at 4:22 PM, Daniel Friesen daniel@nadir-seen-fire.comwrote:
On Tue, 02 Jul 2013 09:30:57 -0700, Jeroen De Dauw jeroendedauw@gmail.com wrote:
- Installing extensions leaves the composer.json file modified.
When installing one or more extensions via Composer, they will get added to the require section in composer.json. composer.json is not in the gitignore list. So you might well run into conflicts here, and in any case will have a modified file that is tracked in your git repo, which is annoying. In case of extension installation via LocalSettings, this stuff is in a file that is in gitignore. We could just add composer.json there as well, but this means that when we make changes to it on master, people will not get them any time soon. This is problematic in case we where to make MW core dependent on other packages, though this seems unlikely to happen, and is thus perhaps just a theoretical concern. Does anyone see a way around this problem better than putting composer.json in gitignore? Any concerns with putting it in gitignore?
This is precisely one of the reasons I thought installing MW extensions via Composer was a stupid idea in the first place. Installing extensions should not require the file that MediaWiki declares it's library dependencies and information about itself to be modified.
We can't be the only people in the world with this problem. Instead of abandoning an excellent attempt by the PHP community to provide alternatives to NIH can we lend some expertise in solving the issue instead?
I do not consider this a case of NIH. Composer is NOT an attempt by the PHP community to provide a way to mange their own plugins. Composer is an attempt by the PHP community to let PHP applications and libraries depend and install 3rd party libraries that they directly need.
The uses and requirements for the two are different they aren't the same thing.
It's not NIH to decide to create your own hat when all you have around you to wear are boots.
Another one is extension assets. Have you dealt with the fact that MW doesn't have a proper path to serve the css/js in debug mode, images to IE, and other assets that an extension needs.
ditto
Erik Bernhardson
On Tue, Jul 2, 2013 at 5:56 PM, Daniel Friesen daniel@nadir-seen-fire.comwrote:
I do not consider this a case of NIH. Composer is NOT an attempt by the PHP community to provide a way to mange their own plugins. Composer is an attempt by the PHP community to let PHP applications and libraries depend and install 3rd party libraries that they directly need.
Apologies if i wasn't clear, I was referring to NIH in a different way. In my mind, after code review for performance/security considerations, managing 3rd party dependencies is a substantial roadblock to using them. The amount of shared code I use in php(outside of MW) has increased quadratically(shrug, why not? :-) since i started using composer. Having tools which simplify the process of ensuring every developer/server/etc. has the same version of those dependencies and the dependencies of their dependencies is a big step towards making it possible to use 3rd party code.
I see the composer project as a tool for removing some of the roadblocks between the developer and their desire to use open source solutions. Between the developer and their desire to have an opportunity through using and contributing back to 3rd party code to not only make life better for MediaWiki, but for the PHP community as a whole.
I also agree composer is not a plugin installer, but why do we need to also invent our own dependency management? The stated issue which you (and by proxy I) replied to is about a file distributed with mediawiki changing when the user wishes to install additional packages. I have seen the same issue with regards to composer brought up in other forums, and am convinced that with the quality of developers working on mediawiki it is a problem we could solve without changing what composer is(i dunno, perhaps allow a second file composer_user.{json,lock} which gets merged with composer.{json,lock}?). I am willing to work on that issue when I have time, but first I will need to communicate with the composer authors to see what direction they think is reasonable.
Erik Bernhardson
Le 03/07/13 04:16, Erik Bernhardson a écrit :
I also agree composer is not a plugin installer, but why do we need to also invent our own dependency management?
That was exactly my idea when I drafted some rough composer support. It has support for custom installers and I wrote a very basic one for MediaWiki extensions https://github.com/composer/installers/pull/37
I am pretty sure Drupal has the same issue with plugins install, any interested people could work with them and Composer to figure out a way to have a local composer.json that will add custom packages not shipped by the software they use (aka MediaWiki).
On Tue, Jul 2, 2013 at 7:22 PM, Daniel Friesen daniel@nadir-seen-fire.comwrote:
Another one is extension assets. Have you dealt with the fact that MW doesn't have a proper path to serve the css/js in debug mode, images to IE, and other assets that an extension needs.
Installing to custom locations is trivial. And if we were to move to a full Composer setup, we'd likely have a custom installer script for MW extensions (which composer has support for).
I do not consider this a case of NIH. Composer is NOT an attempt by the PHP
community to provide a way to mange their own plugins. Composer is an attempt by the PHP community to let PHP applications and libraries depend and install 3rd party libraries that they directly need.
This is probably the more important point. Even Composer's own documentation makes it pretty clear that composer is *not* a package manager. We should be using composer solely for dependency management.
The main issue, though, that I was hoping Composer could solve, was not extension installation but extension dependencies. For example, if there are two or more extensions that require the same dependencies, have only one copy of the 3rd party libraries across all of them. And that is exactly what Composer was made to do, but achieving this is difficult considering extensions are separate projects.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
Hey,
The main issue, though, that I was hoping Composer could solve, was not
extension installation but extension dependencies.
The most common for of extension dependencies are other extensions, so in practice there appears to be little distinction there. If we allow installation of extensions A, B and C since they are needed by extension D, then why would we not allow installing extension D in one go as well? Not much extra effort is needed. There is also not much need to make a distinction between if it is a "MediaWiki extension" or a third party library.
I fully agree with you that resolving these kinds of dependencies is what Composer is for, and that it would be very nice if we could use it to install all dependencies for some extension. That is the main thing I want to get out of Composer. And this is already working to a big extend for some extensions. For instance you can install Wikibase QueryEngine using "composer create-project wikibase/query-engine:dev-master WikibaseQueryEngine --keep-vcs". Which will install its 5 dependencies, including 1 indirect one [0]. When doing so, one still has to include WikibaseQueryEngine in LocalSettings however, since the composer autoloader is not loaded. Furthermore, if done like this, you end up with the dependencies under core/extensions/WikibaseQueryEngine/vendor, and will have them multiple times if you do other such installs in the extensions directory. This is thus not a good way to use Composer for installing extensions within a project such as MW. If we follow one of the approaches Hashar described in his last mail, we do not have these issues.
[0] You can see this working here https://travis-ci.org/wikimedia/mediawiki-extensions-WikibaseQueryEngine/job... the line starting with bash)
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
Le 02/07/13 18:30, Jeroen De Dauw a écrit :
Over the past few months a number of people have been poking at Composer [0] support for MediaWiki [1]. Today I had a look at this and found that though we are close to getting this to work, there are a few remaining problems to be tackled.
<snip>
- MediaWiki needs to load the composer autoloader when present.
- Installing extensions leaves the composer.json file modified.
- Not clear how to best install an extension
<snip>
Hello,
At first, thank you Jeroen from bringing this topic there and for all your investment on Composer =)
== creating a project the composer way ==
I talked a bit with #composer people (there are some available during European business hours). Here is a rough summary of my conversation with them this morning.
Composer being a dependency manager for your project, if you create a wiki locally you should create your own composer.json then require mediawiki/core and your extensions. IE MediaWiki itself would just be a dependency for your site project. So the process would be something like:
mkdir mysite && cd !:1 composer init -n --name=hashar/mysite --stability=dev
// Then install MediaWiki composer require mediawiki/core:dev-REL1_21
// And the abuse filter extension with its dependencies: composer require mediawiki/abuse-filter:dev-REL1_21
You end up with the following hierarchy:
/extensions/AbuseFilter /extensions/AntiSpoof /vendor/composer /vendor/mediawiki/core
The vendor/autoload.php is supposed to be required in your site and you would get everything you need. MediaWiki core does not provide an autoload entry in its composer.json.
== custom bundle ==
I have been pointed to https://github.com/symfony/symfony-standard which is a basic application making use of the Symfony framework. It has a few dependencies one of them being symfony/symfony (the full framework).
If one wanted to update the 'upstream' framework or ship additional dependency, he would just have to edit the symfony-standard composer.json and add them there.
== Composer hooks ==
Seldaek on IRC pointed to Composer hooks such as pre-update-cmd and pre-install-cmd. They could be used in MediaWiki core composer.json to load an extensions.json and inject dependencies on the fly. This way people could git clone mediawiki/core and then edit their local json to add extensions they want.
Some basic doc is at http://getcomposer.org/doc/articles/scripts.md
== Embedded Composer ==
The idea is to get Composer inside your application this way it can fetch the dependencies it needs.
Doc: https://github.com/dflydev/dflydev-embedded-composer#why-would-i-want-to-emb...
Presentation: http://srcmvn.com/blog/2013/05/23/symfony-live-2013-portland-embedded-compos...
If we want to do it the composer way, MediaWiki should be a standalone framework that you install under vendor and don't care much about. Then you could add to your project the extensions you need. Ideally your /index.php would be something like:
<?php require 'vendor/autoload.php' $wgSomeFeature = 'bah';
new MediaWiki()->run();
Hey Hashar,
Thanks for looking into this and sharing all the useful info :)
If we want to do it the composer way, MediaWiki should be a standalone
framework that you install under vendor and don't care much about
We probably ought to create a list of the blockers for this. One item on it is going to be global scope assumptions through core and extension initialization code. Which is unfortunate, as it is probably going to take a while before that one is fully fixed.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
wikitech-l@lists.wikimedia.org