Hey all,
Several MediaWiki extensions, such as Maps and Semantic MediaWiki, are now installable via Composer. Lately I've been getting quite a few questions from developers on how to have their extensions support this awesome dependency manager, and decided to write down the most important points that are not obviously evident in the Composer documentation down.
http://www.bn2vs.com/blog/2013/11/24/introduction-to-composer-for-mediawiki-...
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
Hi Jeroen,
I'd love for someone to take a crack at writing an RFC for this, and maybe we can get this on the agenda for the Architecture Summit. It seems to be a topic that comes up frequently, and has a number of champions, so it doesn't have to be you that does this. Thoughts?
Rob
On Sun, Nov 24, 2013 at 9:10 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
Hey all,
Several MediaWiki extensions, such as Maps and Semantic MediaWiki, are now installable via Composer. Lately I've been getting quite a few questions from developers on how to have their extensions support this awesome dependency manager, and decided to write down the most important points that are not obviously evident in the Composer documentation down.
http://www.bn2vs.com/blog/2013/11/24/introduction-to-composer-for-mediawiki-...
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 -- _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hi everyone,
Jeroen let me know offlist that he's not interested in writing an RFC in this area, so I'm wondering if there are any other takers for Composer use in MediaWiki.
I think there may be multiple RFCs to consider: 1. Composer as a means of MediaWiki extension management 2. Composer as a means of MediaWiki shared library management 3. Exposing certain functionality in embedded in MediaWiki as reusable Composer libraries.
It seems that the idea of using Composer more in MediaWiki generally comes up at least once every 2-3 months, but we never get beyond the "wouldn't it be nice if...." stage. I honestly don't think it would take that much of a shove to put us on a trajectory to using it much more seriously.
Rob
On Fri, Dec 6, 2013 at 11:03 AM, Rob Lanphier robla@wikimedia.org wrote:
Hi Jeroen,
I'd love for someone to take a crack at writing an RFC for this, and maybe we can get this on the agenda for the Architecture Summit. It seems to be a topic that comes up frequently, and has a number of champions, so it doesn't have to be you that does this. Thoughts?
Rob
On Sun, Nov 24, 2013 at 9:10 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
Hey all,
Several MediaWiki extensions, such as Maps and Semantic MediaWiki, are now installable via Composer. Lately I've been getting quite a few questions from developers on how to have their extensions support this awesome dependency manager, and decided to write down the most important points that are not obviously evident in the Composer documentation down.
http://www.bn2vs.com/blog/2013/11/24/introduction-to-composer-for-mediawiki-...
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 -- _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 2013-12-06 5:46 PM, Rob Lanphier wrote:
It seems that the idea of using Composer more in MediaWiki generally comes up at least once every 2-3 months, but we never get beyond the "wouldn't it be nice if...." stage. I honestly don't think it would take that much of a shove to put us on a trajectory to using it much more seriously.
Strictly speaking one thing did get past the "wouldn't it be nice if...." stage. It used to be possible to install PHPUnit and run our tests with that simply by running `{composer} install`, until someone went and deleted composer.json as a "feature". Which also now that I look back has also messed up the implementation of the " Update our code to use RDFa 1.1 instead of RDFa 1.0"[1] RFC since that feature was the first one to require a composer installed library to function (it only needed it to run tests, but it was the first toe dipped into the water).
[1] https://www.mediawiki.org/wiki/Requests_for_comment/Update_our_code_to_use_R...
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
Rob Lanphier wrote:
Jeroen let me know offlist that he's not interested in writing an RFC in this area, so I'm wondering if there are any other takers for Composer use in MediaWiki.
I think there may be multiple RFCs to consider:
- Composer as a means of MediaWiki extension management
- Composer as a means of MediaWiki shared library management
- Exposing certain functionality in embedded in MediaWiki as reusable
Composer libraries.
It seems that the idea of using Composer more in MediaWiki generally comes up at least once every 2-3 months, but we never get beyond the "wouldn't it be nice if...." stage. I honestly don't think it would take that much of a shove to put us on a trajectory to using it much more seriously.
Is adding Composer support orthogonal to the idea of making it trivial to install MediaWiki extensions? The goals seem to be roughly the same: making it easier to safely install new MediaWiki extensions. Presumably a graphical user interface installer would include the ability to do dependency checking, PHP version checking, etc.
MZMcBride
Hey,
Is adding Composer support orthogonal to the idea of making it trivial to
install MediaWiki extensions?
It is not.
Strictly speaking one thing did get past the "wouldn't it be nice
if...." stage. It used to be possible to install PHPUnit and run our tests with that simply by running `{composer} install`, until someone went and deleted composer.json as a "feature".
This comment seems to be based on misunderstanding of both the past and present situation. Let me clarify. Before the mentioned change:
* PHPUnit would be installed by default whenever doing an install via Composer * People could not install anything else via Composer as this would cause conflicts in composer.json
Present situation:
* By default nothing is installed * Users can choose whatever they want to add, _including_ PHPUnit
It seems that the idea of using Composer more in MediaWiki generally comes
up at least once every 2-3 months, but we never get beyond the "wouldn't it be nice if...." stage.
Several extensions currently support installation via Composer, and this capability is already utilized by users of those extensions. For the next release of Semantic MediaWiki, installation via Composer will be the recommended approach. So we are not only past the "wouldn't it be nice if...." stage, we also are past the initial implementation stage. There are still many additional "wouldn't it be nice if...." things that can be done on top, such as creating a GUI for installing extensions. Perhaps writing an RFC for those would be of actual use, though in a lot of cases creating a proof of concept and discussing that might be better use of ones time.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On Sat, Dec 7, 2013 at 8:09 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
Present situation:
- By default nothing is installed
- Users can choose whatever they want to add, _including_ PHPUnit
This is forgetting the point that is the topic of this thread: you cannot currently add dependencies for the MediaWiki core itself because there is no composer.json file to add them to. If I wanted to add a third party library as a dependency to MediaWiki core, how would I do it? (Keep in mind I'm not talking in the scope of my own installation, I mean adding a dependency to core itself through Gerrit.)
Right now the only somewhat proper approach to adding new core dependencies is using Git submodules, which is not the most favorable approach for reasons I mentioned in the beginning of the thread.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science
On Sun, Dec 8, 2013 at 2:20 PM, Tyler Romeo tylerromeo@gmail.com wrote:
On Sat, Dec 7, 2013 at 8:09 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
Present situation:
- By default nothing is installed
- Users can choose whatever they want to add, _including_ PHPUnit
This is forgetting the point that is the topic of this thread: you cannot currently add dependencies for the MediaWiki core itself because there is no composer.json file to add them to. If I wanted to add a third party library as a dependency to MediaWiki core, how would I do it? (Keep in mind I'm not talking in the scope of my own installation, > I mean adding a dependency to core itself through Gerrit.)
Right now the only somewhat proper approach to adding new core dependencies is using Git submodules, which is not the most favorable approach for reasons I mentioned in the beginning of the thread.
I've been thinking about this problem on and off for a while. I think that one of the issues that is being bumped into with using Composer in core is that core is both a library (or really collection of libraries) and an application.
I believe that the composer.json file was taken out of core to make it easier for consumers of MW the application to use composer to manage extensions in their local install via composer. This supports Joroen's use case of allowing folks running a MW instance to manage their extensions using Composer.
The side effect is that it removed the ability to use Composer to manage external components used by MW the library which is Tyler's proposed use case [0].
I have used Composer with both applications [1] and libraries [2], but I've never tried to make a single project serve both roles. Finding a way to separate MW the library from MW the application may be a solution to this conflict. I don't think this would be a trivial project, but it doesn't seem impossible either. There would be several installation use cases to work through for a successful conversion: Foundation wiki farm, developer installs, consumer install via packagist, consumer install via tarball, etc.
[0]: https://www.mediawiki.org/wiki/Requests_for_comment/Third-party_components [1]: https://git.wikimedia.org/summary/wikimedia%2Fwikimania-scholarships [2]: https://packagist.org/users/bd808/
Bryan
Hey,
Finding a way to separate MW the library from MW the application may be a
solution to this conflict. I don't think this would be a trivial project, but it doesn't seem impossible either.
That'd be fanatic if it happened for many other reasons as well. For all intents and purposes it is a big caste in the sky though. I expect this to not happen in the coming years, unless there is a big shift in opinion on what constitutes good software design and architecture in the community.
The side effect is that it removed the ability to use Composer to
manage external components used by MW the library which is Tyler's proposed use case [0].
If the core community actually gets to a point where potential usage of third party libraries via Composer is actually taken seriously, this will indeed need to be tackled. I do not think we are quite there yet. For instance, if we go down this road, getting a clone of MW will no longer be sufficient to have your wiki run, as some libraries will first need to be obtained. This forces change to very basic workflow. People who dislike Composer will thus scream murder. Hence I tend to regard this as a moot point for now. Though lets pretend this has already bee taken care off and look at solutions to the technical problem.
One approach, that is a lot more feasible then making MediaWiki (partially) library like, is to specify the MW dependencies somewhere else. Not in composer.json. Then when you install MediaWiki, these dependencies get added automatically to composer.json. And when you update it, new ones also get added. In a way, this is similar to the generation of LocalSettings. This is the approach I'd be investigating further if we actual where at a point where a technical solution is needed.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On Mon, Dec 9, 2013 at 12:06 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
Finding a way to separate MW the library from MW the application may be a
solution to this conflict. I don't think this would be a trivial project, but it doesn't seem impossible either.
That'd be fanatic if it happened for many other reasons as well. For all intents and purposes it is a big caste in the sky though. I expect this to not happen in the coming years, unless there is a big shift in opinion on what constitutes good software design and architecture in the community.
The side effect is that it removed the ability to use Composer to
manage external components used by MW the library which is Tyler's proposed use case [0].
If the core community actually gets to a point where potential usage of third party libraries via Composer is actually taken seriously, this will indeed need to be tackled. I do not think we are quite there yet. For instance, if we go down this road, getting a clone of MW will no longer be sufficient to have your wiki run, as some libraries will first need to be obtained. This forces change to very basic workflow. People who dislike Composer will thus scream murder. Hence I tend to regard this as a moot point for now. Though lets pretend this has already bee taken care off and look at solutions to the technical problem.
One approach, that is a lot more feasible then making MediaWiki (partially) library like, is to specify the MW dependencies somewhere else. Not in composer.json. Then when you install MediaWiki, these dependencies get added automatically to composer.json. And when you update it, new ones also get added. In a way, this is similar to the generation of LocalSettings. This is the approach I'd be investigating further if we actual where at a point where a technical solution is needed.
No one has any major objections to composer *support*. People have objections to composer being required, though, as many of us use git as our method of installation and deployment and have very valid beliefs that composer is inferior to git with submodules for a number of reasons.
I think the biggest obstacle so far is the half assed way things are currently happening, from all fronts. There's no real support from Wikimedia in this regard. There's a "composer is the right way of do it, and you're an idiot for not using" attitude from some. There's also a relatively large lack of understanding of how we can use this for both extensions and MediaWiki all around.
Someone needs to actually take the reigns on this, or let's stop wasting the list's time with it.
- Ryan
On Sun, Dec 8, 2013 at 6:06 PM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
If the core community actually gets to a point where potential usage of third party libraries via Composer is actually taken seriously, this will indeed need to be tackled. I do not think we are quite there yet. For instance, if we go down this road, getting a clone of MW will no longer be sufficient to have your wiki run, as some libraries will first need to be obtained. This forces change to very basic workflow. People who dislike Composer will thus scream murder. Hence I tend to regard this as a moot point for now. Though lets pretend this has already bee taken care off and look at solutions to the technical problem.
OK, in that case I will use git submodules for my upcoming third party library patches.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science
On 2013-12-08 3:06 PM, Jeroen De Dauw wrote:
The side effect is that it removed the ability to use Composer to manage external components used by MW the library which is Tyler's proposed use case [0].
If the core community actually gets to a point where potential usage of third party libraries via Composer is actually taken seriously, this will indeed need to be tackled. I do not think we are quite there yet.
mediawiki-l has had a topic on UserMailer, we already have an RFC on starting to use 3rd party components inside MediaWiki[1], and I already had a commit (the RDFa code) happily using a composer installed library which is naturally broken now that composer.json is gone. This is *precisely* the time handling library dependencies using composer is relevant and why breaking the ability to manage libraries with composer is a problem. Frankly I find it absurd to suggest that the idea of installing libraries with composer needs to prove the community takes it seriously but the idea of installing extensions with composer doesn't need to prove it before jumping in and becoming a roadblock in being able to manage libraries with composer.
[1] https://www.mediawiki.org/wiki/Requests_for_comment/Third-party_components
For instance, if we go down this road, getting a clone of MW will no longer be sufficient to have your wiki run, as some libraries will first need to be obtained. This forces change to very basic workflow. People who dislike Composer will thus scream murder. Hence I tend to regard this as a moot point for now. Though lets pretend this has already bee taken care off and look at solutions to the technical problem.
You're supposing the existence of a notable group I haven't seen before as if it were a big part of the community. In all our prior discussions on composer I don't recall seeing anyone coming out saying they hated the idea of using composer for libraries. The only people opposing composer those time I recall were people like me who only hated the idea of using composer for extensions, finding the idea absolutely absurd and a complete misuse of composer.
In any case this suggestion that we have people that hate composer enough we can't use it as part of our primary workflow is a complete red herring. Package managers like npm, bower, and composer that install packages locally can have their dependencies directly committed into source control. ie: Composer can still be used to manage dependencies and upgrade libraries but because the dependencies are committed no-one is required to run `{composer} install` after clone. Composer likes to dissuade you from this more than other package managers, but it is perfectly possible and reasonable to do if such a group was an issue. So even if we did have a group like that, it's completely irrelevant, a distracting red herring, not something that makes the real issue a moot point.
And we don't even need a big switchover to composer before blocking the ability to manage dependencies with composer.json becomes an issue. Using Composer for optional dependencies like PHPUnit and libraries that are only needed for specific features or only needed to do a full test run are the perfect way to dip MediaWiki's toes into composer and start getting things ready for composer to be used for more important things. I had *already* started doing this. PHPUnit was fully supported by composer. I had a commit for an RFC people liked using a composer installed 3rd party library for an optional set of tests. If that had gone through before composer.json was deleted we would already be having the discussion on how to handle composer dependencies in jenkins.
One approach, that is a lot more feasible then making MediaWiki (partially) library like, is to specify the MW dependencies somewhere else. Not in composer.json. Then when you install MediaWiki, these dependencies get added automatically to composer.json. And when you update it, new ones also get added. In a way, this is similar to the generation of LocalSettings. This is the approach I'd be investigating further if we actual where at a point where a technical solution is needed.
I'd like to note that this precludes the ability to add composer installed dependencies directly to git.
But more importantly, frankly the fact that we're talking about ways to avoid having a composer.json file committed to git like the standard for every other repo using composer is one of either: A) Another assertion that trying to install extensions via composer is a complete abuse and misuse of composer. B) A flaw in composer itself, which you may want to consider having fixed before you go installing extensions with it. Because composer is the *only* package manager I have seen that requires that all installed packages must be specified in the whatever.json file. Even npm and bower which work just like composer, installing libraries in a local folder, do not require that you add everything to package.json or bower.json.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
On Sun, Dec 8, 2013 at 3:06 PM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
On Sun, Dec 8, 2013 at 2:42 PM, Bryan Davis bd808@wikimedia.org wrote:
Finding a way to separate MW the library from MW the application may be a solution to this conflict. I don't think this would be a trivial project, but it doesn't seem impossible either.
That'd be fanatic if it happened for many other reasons as well. For all intents and purposes it is a big caste in the sky though. I expect this to not happen in the coming years, unless there is a big shift in opinion on what constitutes good software design and architecture in the community.
It sounds like you're retreating from an argument you haven't even started yet. A successful proposal will likely be one that can be executed incrementally without huge workflow shifts, so there may have been resistance in the past to a particular "blow it all up and start over" strategy. However, I haven't yet heard anyone put forward the argument that MediaWiki's monolithic architecture is the correct long-term architecture.
I largely agree with Bryan that separating MediaWiki the library from MediaWiki the application is something we should consider tackling. I have a quibble with the wording: it should be "MediaWiki the set of libraries", not a single library, and it seems like we could start teasing them out one at a time if we can agree on a general framework under which this would be done.
Rob
Sent from my iPhone
On Dec 8, 2013, at 10:22 PM, Rob Lanphier robla@wikimedia.org wrote:
On Sun, Dec 8, 2013 at 3:06 PM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
On Sun, Dec 8, 2013 at 2:42 PM, Bryan Davis bd808@wikimedia.org wrote:
Finding a way to separate MW the library from MW the application may be a solution to this conflict. I don't think this would be a trivial project, but it doesn't seem impossible either.
That'd be fanatic if it happened for many other reasons as well. For all intents and purposes it is a big caste in the sky though. I expect this to not happen in the coming years, unless there is a big shift in opinion on what constitutes good software design and architecture in the community.
It sounds like you're retreating from an argument you haven't even started yet. A successful proposal will likely be one that can be executed incrementally without huge workflow shifts, so there may have been resistance in the past to a particular "blow it all up and start over" strategy. However, I haven't yet heard anyone put forward the argument that MediaWiki's monolithic architecture is the correct long-term architecture.
I largely agree with Bryan that separating MediaWiki the library from MediaWiki the application is something we should consider tackling. I have a quibble with the wording: it should be "MediaWiki the set of libraries", not a single library, and it seems like we could start teasing them out one at a time if we can agree on a general framework under which this would be done.
Rob _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Not to be a downer but isn't this one of the things x windows did that upset people? I'm not arguing that this approach is doomed, just that care must be taken. Honestly I don't know the situation well enough to have a super strong opinion.
Nik
On Sun, Dec 8, 2013 at 10:43 PM, Nik Everett neverett@wikimedia.org wrote:
Not to be a downer but isn't this one of the things x windows did that upset people? I'm not arguing that this approach is doomed, just that care must be taken. Honestly I don't know the situation well enough to have a super strong opinion.
Agreed on being very hesitant with this. Not *all* of MediaWiki needs to be a library. A library implies re-use. For example, nobody is going to reuse the SpecialPage class and its children outside of MediaWiki, so it does not make sense to turn it into a library.
The only things that could really be separated into libraries are the DatabaseType hierarchy, the HTMLForm section, maybe the FileRepo / File / FileBackend product, and maybe the JobQueue product.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science
Hey,
On 9 December 2013 04:22, Rob Lanphier robla@wikimedia.org wrote:
That'd be fanatic if it happened for many other reasons as well. For all intents and purposes it is a big caste in the sky though. I expect this
to
not happen in the coming years, unless there is a big shift in opinion on what constitutes good software design and architecture in the community.
It sounds like you're retreating from an argument you haven't even started
yet.
I outlined something I expect to happen, a prediction of the probable future. It is not an argument, not a suggestion, not a wish and not something I'm saying should be done. Do you disagree it is unlikely we will get to the point where MediaWiki can behave library like (where applicable) in the near future?
On 9 December 2013 04:22, Rob Lanphier robla@wikimedia.org wrote:
A successful proposal will likely be one that can be executed incrementally without huge workflow shifts, so there may have been resistance in the past to a particular "blow it all up and start over" strategy.
Exactly. Incrementalism is generally the way to go. See also: agile. I for one think it is a very bad idea to rewrite huge chunks of a codebase at a time, due to high risk, high cost and sometimes plain infeasibility. Though note how that there are many people that tend to scream "why did you not fix the whole of x" whenever an incremental solution is created, or that demand one goes all the way when it is proposed.
On 9 December 2013 04:22, Rob Lanphier robla@wikimedia.org wrote:
However, I haven't yet heard anyone put forward the argument that MediaWiki's monolithic architecture is the correct long-term architecture.
The argument is definitely there. People that take this point of view are obviously not phrasing it as "I like big monolithic architectures" though.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
wikitech-l@lists.wikimedia.org