I attempted to install Wikibase the other day and made a fun discovery. Installing it properly requires the following (12) extensions:
WikibaseClient Wikibase DataModel WikibaseLib Wikibase Repository DataValues DataTypes ValueParsers ValueView ValueValidators ValueFormatters Diff Scribunto
And the following optional (2) extensions:
Universal Language Selector Babel
Soon it will also require at least the following (4) extensions:
Ask Wikibase Query Wikibase Database Wikibase QueryEngine
To fully deploy wikibase in a way that will work like wikidata, it will take at least 18 extensions, all of which are versioned differently, and are broken out in this manner to be used as libraries.
What this is subtly doing is adding another dependency chain into MediaWiki: extension libraries. Since these extensions are meant to be used as libraries, other extensions will eventually do so and admins will have to worry about not only extension compatibility with MediaWiki (an already nearly impossible task), but will also need to worry about extension dependency with extension libraries. The compatibility matrix for this is going to be terrible and exacerbates one of MediaWiki's biggest problems for admins.
Quite a few of these should be core functionality or if they can't properly pass review they should be removed. For legitimate library-like extensions I have no constructive alternative, but there must be some sane alternative to this.
- Ryan
It might also be worth going a "wikibase" release (on its own schedule or else scheduled shortly after each MW release), that contains a particular MW version along with version of the extensions that have been tested to work with it. This is the usual way in which these sorts of dependency chains are dealt with outside of WMF. (Ie, when RedHat Fedora is released, it contains the latest GNOME at that time, which in turn contains a version of GTK+ and various utilities all tested to work well together).
For the admin, it should just be a matter of using "wikibase 1.23" or whatever; they shouldn't have to hunt down individual extensions and figure out compatible versions for themselves. --scott
On Fri, Jul 19, 2013 at 11:20 AM, C. Scott Ananian cananian@wikimedia.orgwrote:
It might also be worth going a "wikibase" release (on its own schedule or else scheduled shortly after each MW release), that contains a particular MW version along with version of the extensions that have been tested to work with it. This is the usual way in which these sorts of dependency chains are dealt with outside of WMF. (Ie, when RedHat Fedora is released, it contains the latest GNOME at that time, which in turn contains a version of GTK+ and various utilities all tested to work well together).
For the admin, it should just be a matter of using "wikibase 1.23" or
whatever; they shouldn't have to hunt down individual extensions and figure out compatible versions for themselves.
What if you want to use wikibase 1.23 and MyAbominationExtension 1.5 that requires an incompatible version of DataValues and MyAwesomeExtension 1.0 that requires an incompatible version of ValueView?
- Ryan
Hey,
What if you want to use wikibase 1.23 and MyAbominationExtension 1.5 that
requires an incompatible version of DataValues and MyAwesomeExtension 1.0 that requires an incompatible version of ValueView?
If you have releases of certain software that have requirements that cannot satisfied together, then you cannot install them together. That is a pretty inherent property of incompatible software.
As a user, when I run into such a situation, what I want to know is which versions of the software I am interested in I can install together. That is, after being told the latest releases do not work together. Sounds like we need some kind of package management :) In case of the components created for Wikidata, we have been supporting Composer for a while now, which is a great fit to our needs.
I attempted to install Wikibase the other day and made a fun discovery.
Installing it properly requires the following (12) extensions:
That is somewhat inaccurate, and is misleading with regard to Wikibase installation. Nevertheless, the concerns you bring up are certainly relevant, and currently not really tackled well in the MediaWiki community. That is to bad, as it encourages people to inappropriately bundle things and throw re usability out of the window (plus causing a long list of other problems).
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
Man, if only PHP had some sort of dependency management system.....
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
On Fri, Jul 19, 2013 at 2:44 PM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
Hey,
What if you want to use wikibase 1.23 and MyAbominationExtension 1.5 that
requires an incompatible version of DataValues and MyAwesomeExtension 1.0 that requires an incompatible version of ValueView?
If you have releases of certain software that have requirements that cannot satisfied together, then you cannot install them together. That is a pretty inherent property of incompatible software.
As a user, when I run into such a situation, what I want to know is which versions of the software I am interested in I can install together. That is, after being told the latest releases do not work together. Sounds like we need some kind of package management :) In case of the components created for Wikidata, we have been supporting Composer for a while now, which is a great fit to our needs.
I attempted to install Wikibase the other day and made a fun discovery.
Installing it properly requires the following (12) extensions:
That is somewhat inaccurate, and is misleading with regard to Wikibase installation. Nevertheless, the concerns you bring up are certainly relevant, and currently not really tackled well in the MediaWiki community. That is to bad, as it encourages people to inappropriately bundle things and throw re usability out of the window (plus causing a long list of other problems).
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 -- _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 07/19/2013 02:46 PM, Tyler Romeo wrote:
Man, if only PHP had some sort of dependency management system.
The real question is: why doesn't MediaWiki use Composer? http://getcomposer.org/download/
We've discussed this here before, though, so it really isn't anything more than a rhetorical question.
Ryan's post just shows us, once again, why there is a need for it.
On Fri, Jul 19, 2013 at 3:16 PM, Mark A. Hershberger mah@everybody.orgwrote:
The real question is: why doesn't MediaWiki use Composer? http://getcomposer.org/download/
We've discussed this here before, though, so it really isn't anything more than a rhetorical question.
Ryan's post just shows us, once again, why there is a need for it.
From what Antoine found out from the composer people, basically what we'll
have to do to make MW use Composer is that we need to separate the entry points from the backend classes (in other words, put everything except index.php, api.php, etc. in one repository, and then have the entry points in another). That way the MW core itself becomes a library, which is how Symfony does it. Then people just make a project from the entry point project, and then they can use "composer require" to add more extensions.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
Le 19/07/13 22:28, Tyler Romeo a écrit :
From what Antoine found out from the composer people, basically what we'll have to do to make MW use Composer is that we need to separate the entry points from the backend classes (in other words, put everything except index.php, api.php, etc. in one repository, and then have the entry points in another). That way the MW core itself becomes a library, which is how Symfony does it. Then people just make a project from the entry point project, and then they can use "composer require" to add more extensions.
Exactly :-]
And installing yoursite would be something like:
mkdir mysite cd mysite composer require mediawiki/core composer require wikibase/wikibase # which installs data-values/data-values ask/ask as well
Your root directory ends up with a composer.json:
{ "require": { "mediawiki/core": "*", "diff/diff": "*", "wikibase/wikibase": "*" } }
And a vendor directory containing all dependencies. At the root of your directory you would write some glue maybe looking like:
<?php require('vendor/autoload.php'); require('vendor/mediawiki/core/index.php');
(that does not work, but you get the idea).
2013/7/19 Antoine Musso hashar+wmf@free.fr
And installing yoursite would be something like:
mkdir mysite cd mysite composer require mediawiki/core composer require wikibase/wikibase # which installs data-values/data-values ask/ask as well
Just curious, why is "composer require mediawiki/core" needed?
Hey,
Just curious, why is "composer require mediawiki/core" needed?
Because Hashar is talking about having a "MediaWiki installation" package, which effectively turns into "My MediaWiki installation" when you start using it. This package specifies all the things you want to install (in its composer.json). This will always include core, and also list the extensions you want on your particular install.
The reason for having this seperate package is to get around the problem I described earlier in this thread - if you just have core, and you install other things into it by modifying the composer.josn of core, you'll end up with a modified non-optional file in the core repo, which causes hassle when you want to update.
Of course if we got with such a "MediaWiki installation" package, I'd be natural to put mediawiki-core already in the composer.json file, so people would not actually need to run this command.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On 22.07.2013, 16:11 Jeroen wrote:
Hey,
Just curious, why is "composer require mediawiki/core" needed?
Because Hashar is talking about having a "MediaWiki installation" package, which effectively turns into "My MediaWiki installation" when you start using it. This package specifies all the things you want to install (in its composer.json). This will always include core, and also list the extensions you want on your particular install.
The reason for having this seperate package is to get around the problem I described earlier in this thread - if you just have core, and you install other things into it by modifying the composer.josn of core, you'll end up with a modified non-optional file in the core repo, which causes hassle when you want to update.
Of course if we got with such a "MediaWiki installation" package, I'd be natural to put mediawiki-core already in the composer.json file, so people would not actually need to run this command.
So it's a Composer's limitation. It woud've been awesome if it was possible to do something like
if exists "optional.json": install "optional.json"
:)
Le 22/07/13 13:29, Denny Vrandečić a écrit :
And installing yoursite would be something like:
mkdir mysite cd mysite composer require mediawiki/core composer require wikibase/wikibase # which installs data-values/data-values ask/ask as well
Just curious, why is "composer require mediawiki/core" needed?
wikibase/wikibase does not list mediawiki/core as a dependency :)
$ composer show wikibase/wikibase ... requires: php >=5.3.0 data-values/data-values * wikibase/data-model * diff/diff >=0.6 data-values/data-types *
Hey,
wikibase/wikibase does not list mediawiki/core as a dependency :)
Indeed. Right now this just allows you to install Wikibase into an existing MW install. Before we can go all the way, we first need to be able to do a MediaWiki (+extensions) install, which is something still under discussion.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
Ah, OK, understood. Thanks.
2013/7/22 Jeroen De Dauw jeroendedauw@gmail.com
Hey,
wikibase/wikibase does not list mediawiki/core as a dependency :)
Indeed. Right now this just allows you to install Wikibase into an existing MW install. Before we can go all the way, we first need to be able to do a MediaWiki (+extensions) install, which is something still under discussion.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 -- _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 19/07/13 23:26, Antoine Musso a écrit :
mkdir mysite cd mysite composer require mediawiki/core composer require wikibase/wikibase # which installs data-values/data-values ask/ask as well
<snip>
And a vendor directory containing all dependencies. At the root of your directory you would write some glue maybe looking like:
<?php require('vendor/autoload.php'); require('vendor/mediawiki/core/index.php');
The glue need to give a hint to WebStart to properly initialize $IP
require('vendor/autoload.php'); putenv( 'MW_INSTALL_PATH=' . __DIR__ . '/vendor/mediawiki/core' ); require('vendor/mediawiki/core/index.php');
:)
On Fri, Jul 19, 2013 at 11:44 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
Hey,
What if you want to use wikibase 1.23 and MyAbominationExtension 1.5 that
requires an incompatible version of DataValues and MyAwesomeExtension 1.0 that requires an incompatible version of ValueView?
If you have releases of certain software that have requirements that cannot satisfied together, then you cannot install them together. That is a pretty inherent property of incompatible software.
Yes, but the only requirement right now is "Am I using the correct version of this extension with this version of MediaWiki?" and you're adding in a whole new set of incompatibilities. You're really not thinking of this from the perspective of the person using the software.
As a user, when I run into such a situation, what I want to know is which versions of the software I am interested in I can install together. That is, after being told the latest releases do not work together. Sounds like we need some kind of package management :) In case of the components created for Wikidata, we have been supporting Composer for a while now, which is a great fit to our needs.
"We" in this situation is Wikidata and not the developer community. In fact, there were a number of threads about composer with no consensus and a number of objections. So, what you're saying is that the Wikidata team has made a decision on behalf of the community?
I attempted to install Wikibase the other day and made a fun discovery.
Installing it properly requires the following (12) extensions:
That is somewhat inaccurate, and is misleading with regard to Wikibase installation. Nevertheless, the concerns you bring up are certainly relevant, and currently not really tackled well in the MediaWiki community. That is to bad, as it encourages people to inappropriately bundle things and throw re usability out of the window (plus causing a long list of other problems).
The solution to this isn't to sneak a requirement in without consensus....
Also, it's poor architecture to split things into a large number of small pieces in anticipation of reuse. It's better to split things apart when there's a need.
Something that's being sidestepped here is that extensions are being used as a means to avoid getting things reviewed for core. Quite a few of these extensions should just be core functionality or they shouldn't exist.
- Ryan
Hey,
you're adding in a whole new set of incompatibilities.
How so?
You're really not thinking of this from the perspective of the person
using the software.
Oh, glad to know you understand what I am thinking, since clearly I do not.
In case of the components
created for Wikidata, we have been supporting Composer for a while now, which is a great fit to our needs.
"We" in this situation is Wikidata and not the developer community. In fact, there were a number of threads about composer with no consensus and a number of objections.
Given my first sentence, I would think it is indeed abundantly clear this is about the components created for Wikidata and not the whole community. Thanks for making it even more obvious.
So, what you're saying is that the Wikidata team has
made a decision on behalf of the community?
What I am saying is that I have played around with something that might be of use to the community in general. Where did I imply I, or the WD team, had made a decision for the whole community here? I don't see it, but since you know better then me what I am thinking, please explain.
Something that's being sidestepped here is that extensions are being used
as a means to avoid getting things reviewed for core. Quite a few of these extensions should just be core functionality or they shouldn't exist.
This is preposterous, and I find the accusation you made outrageous. I'd love to have some constructive discussion here, though this is very difficult if you come at it from the angle you are.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On Sat, Jul 20, 2013 at 4:34 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
Hey,
you're adding in a whole new set of incompatibilities.
How so?
Extensions that use any of these extension libaries now depend on the version of MediaWiki and the version of the extension. That's a new set of dependencies that can be incompatible now.
You're really not thinking of this from the perspective of the person
using the software.
Oh, glad to know you understand what I am thinking, since clearly I do not.
Maintaining MediaWiki installs is already relatively difficult, due to a lack of extension management. Many extensions are also poorly maintained, and just getting an extension that works with your version of MediaWiki reliably is hard. Upgrades are incredibly hard due to this. Adding in another set of incompatibilities is going to make this much harder.
This is making something easier for developers at the expense of admins.
In case of the components
created for Wikidata, we have been supporting Composer for a while now, which is a great fit to our needs.
"We" in this situation is Wikidata and not the developer community. In fact, there were a number of threads about composer with no consensus
and a
number of objections.
Given my first sentence, I would think it is indeed abundantly clear this is about the components created for Wikidata and not the whole community. Thanks for making it even more obvious.
Many of these components are generic enough to
So, what you're saying is that the Wikidata team has
made a decision on behalf of the community?
What I am saying is that I have played around with something that might be of use to the community in general. Where did I imply I, or the WD team, had made a decision for the whole community here? I don't see it, but since you know better then me what I am thinking, please explain.
My concern is adoption of these extension libraries. Other extensions will use these libraries, which is the point of them... If the only way to sanely install them is via composer and a decent number of extensions now relies on them, it's be come a de-facto requirement of MediaWiki, without consensus.
Something that's being sidestepped here is that extensions are being used
as a means to avoid getting things reviewed for core. Quite a few of
these
extensions should just be core functionality or they shouldn't exist.
This is preposterous, and I find the accusation you made outrageous. I'd love to have some constructive discussion here, though this is very difficult if you come at it from the angle you are.
Sorry, I phrased that poorly. Here's my concern: If extension libraries are generic enough that they could be considered core (Diff is a great example), then other extensions will likely use it like core functionality. Wikibase already is. These extensions won't get the same level of review as they would if they were core functionality.
Is there any compelling reason that they are extensions, rather than being added to core?
- Ryan
On Sat, Jul 20, 2013 at 2:08 PM, Ryan Lane rlane32@gmail.com wrote:
In case of the components
created for Wikidata, we have been supporting Composer for a while
now,
which is a great fit to our needs.
"We" in this situation is Wikidata and not the developer community. In fact, there were a number of threads about composer with no consensus
and a
number of objections.
Given my first sentence, I would think it is indeed abundantly clear this is about the components created for Wikidata and not the whole community. Thanks for making it even more obvious.
Many of these components are generic enough to
That's what I get for writing that email out of order... I meant to remove that as I made my point regarding generic components and composer becoming a de-facto standard elsewhere.
- Ryan
Hey,
you're adding in a whole new set of incompatibilities.
How so?
Extensions that use any of these extension libaries now depend on the version of MediaWiki and the version of the extension. That's a new set of dependencies that can be incompatible now.
It adds a component as dependency yes. It however does not add code as dependency. One of the main reasons for creating well designed components rather then throwing everything into one bucket is that it minimizes source code dependencies. If you minimize dependencies (as a good programmer should do on all levels), you also minimize the things that can cause compatibility conflicts. It thus certainly benefits the user.
Imagine you have a package P, which contains components A, B and C. Only package P is versioned and released. That it could be split into 3 packages is a detail not visible to the end user. A is stand alone, B depends on A and C depends on A. You want to install extension E0 that depends on P v1.0 or later. In particular, it depends on B, though again, the end user does not know this. You want to make this install on a wiki where you already have E1 installed. E1 depends on P v0.5 to v0.8. Internally it only cares about C. This situation means you can in fact not install E0 and E1 at the same time. The only reason for this is because to many things are packaged together. If B and C where in their own packages, there would be no issue.
So one can for instance go from depending on package ABC that contains sub components A, B and C to depending on A and B. That's two packages instead of one if you look at it naively, though two components instead of 3 on closer investigation. It is quite bad to force users to care about C while there is no reason for doing so. They'll be affected by any issues that occur in C for no good reason. Every time ABC has a new release because something in C had to change, users will have to deal with this release, since to them it is not visible this is pointless.
These are simple examples, with one component that could be split into 3. The problems for the user get significantly worse if one throws more things together and as this problem is replicated in the dependency chain.
Maintaining MediaWiki installs is already relatively difficult, due to a
lack of extension management. Many extensions are also poorly maintained, and just getting an extension that works with your version of MediaWiki reliably is hard. Upgrades are incredibly hard due to this. Adding in another set of incompatibilities is going to make this much harder.
Like noted above, it is simply not the case more things become dependent on each other. Rather the opposite.
Another example. Take the Diff library. It does not depend on anything (except of course PHP itself). You talk about putting it into core. Imagine Diff 0.7 with MW 1.23 and Diff 1.0 gets bundled with MW 1.24. Now I have extension E0 that needs MW 1.24 or later. I also have extension E1 that needs Diff < 1.0. Again the user runs into problems. In case Diff is merely bundled with core, the user can resolve this by changing either Diff or MW version. In case they are solidly glued together, installation is again not possible for no good reason. Like the one before, this really is a toy example, that merely illustrates how problems arise. The problems users encounter in the real world are more significant.
Now you might ask, where are those problems you talk about?
just getting an extension that works with your version of MediaWiki
reliably is hard.
And finding a set of extensions that work together with that version is even harder. And you'll have hassle whenever you upgrade. I've seen this many many times, and have of course encountered it myself more then enough. After years of listening to users reporting problems and clients running into this, I have concluded that a lot of this could be avoided if people did not introduce silly dependencies.
The Diff extension is a good example how how to get it right. Initially it did not. It dependent on MediaWiki. It was throwing MWExeptions at places, its tests derives from MediaWikiTestCase and whatnot. This bound it to MW, putting in a requirement people have to deal with. This dependency has now been removed, as there really was no reason for it, thus making usage of Diff easier. Once again, this is the simplest example I could think of.
Sorry, I phrased that poorly. Here's my concern: If extension libraries are
generic enough that they could be considered core (Diff is a great example), then other extensions will likely use it like core functionality. Wikibase already is.
I already outlined how this can be problematic, depending on what you mean. If you are suggesting to merge it into core somehow, and just having one release for the whole thing, then it will cause hassle. Bundling would be better. We are already doing this with some extensions. That is something to look at on a per extension or per library basis though. Clearly we do not want to bundle every library needed for any extension with core.
In fact, we can't do this if you consider usage of third party libraries. Imagine you have this app A that uses some common library such as Monolog, and you have two delivery mechanism plugins for this app A. First you have B that is an interface to MW, second you have C that is an interface to some other system. An attempt to "assimilate" A into core is going very likely to end badly. The MW community is not doing a fantastic job when it comes to interoperability and sharing of components with the rest of the PHP world as it is. To me this is a quite unfortunate miss of opportunities.
These extensions won't get the same level of review as they would if they were core functionality.
This implies you are thinking of more then just bundling. Or not?
This is again something that will need judgement on case-by-case basis. In case for Diff, the code quality requirements are significantly higher then those of core, and these are not going to be lowered.
Is there any compelling reason that they are extensions, rather than being
added to core?
Hell yeah! See above :)
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On 21 July 2013 11:30, Jeroen De Dauw jeroendedauw@gmail.com wrote:
Hey,
you're adding in a whole new set of incompatibilities.
How so?
Extensions that use any of these extension libaries now depend on the version of MediaWiki and the version of the extension. That's a new set
of
dependencies that can be incompatible now.
It adds a component as dependency yes. It however does not add code as dependency. One of the main reasons for creating well designed components rather then throwing everything into one bucket is that it minimizes source code dependencies. If you minimize dependencies (as a good programmer should do on all levels), you also minimize the things that can cause compatibility conflicts. It thus certainly benefits the user.
Imagine you have a package P, which contains components A, B and C. Only package P is versioned and released. That it could be split into 3 packages is a detail not visible to the end user. A is stand alone, B depends on A and C depends on A. You want to install extension E0 that depends on P v1.0 or later. In particular, it depends on B, though again, the end user does not know this. You want to make this install on a wiki where you already have E1 installed. E1 depends on P v0.5 to v0.8. Internally it only cares about C. This situation means you can in fact not install E0 and E1 at the same time. The only reason for this is because to many things are packaged together. If B and C where in their own packages, there would be no issue.
So one can for instance go from depending on package ABC that contains sub components A, B and C to depending on A and B. That's two packages instead of one if you look at it naively, though two components instead of 3 on closer investigation. It is quite bad to force users to care about C while there is no reason for doing so. They'll be affected by any issues that occur in C for no good reason. Every time ABC has a new release because something in C had to change, users will have to deal with this release, since to them it is not visible this is pointless.
These are simple examples, with one component that could be split into 3. The problems for the user get significantly worse if one throws more things together and as this problem is replicated in the dependency chain.
The fact that this thought experiment is almost impossible to visualise without pen and paper is symptomatic of the problem. The end user of MediaWiki doesn't care about arcane internal dependencies, or the rationale for them, they want it to Just Work. So either the system has to actually * be* simple (version X of an extension requires version Y of MediaWiki, so if I update my MW install I need to also update extensions (or vice versa)); or it needs to be *properly* abstracted behind a system which makes it *appear* simple (you run some script and everything magically works or gives you informative explanations of why it didn't).
The former is obviously preferable in terms of clarity and maintainability; and that is realised immediately if all essential components are properly backwards-compatible. There should never *be* a dependency of the form "package X *no later than* vY". That is a fault of package X, not of the extension which depends on it. Package X should have been properly managed so that its public signature does not change incompatibly; and if a change is necessary, the change is managed over several versions so that consumers can safely update their use of the new signature and increase their dependency version to the one which introduces the new behaviour. "I need all my components to be sufficiently recent that they support all the features I want" is a methodology that end users can understand (and also one which means that they should *always* be able to get the feature set they want). "I need to calculate the intersection of the dependencies of all my features and resolve them manually or automatically" is not; especially when there may not even *be* an intersection.
None of that is in any way groundbreaking, it's just basic design principles which I think we all subscribe to, even if they don't always materialise. The point is that MediaWiki is *much* more likely to be able to police and maintain that proper signature management in core code, than in extensions and obscure miscellaneous modules. It's great that modules like Diff have good beady eyes on code quality and portability, long may that continue. That is *definitely* the exception, not the rule, with extensions. Our core code definitely isn't perfect in terms of signatures, or even acceptable in many places. But it's getting better because that's where the most attention is focused.
--HM
Hey,
It's great that modules
like Diff have good beady eyes on code quality and portability, long may that continue. That is *definitely* the exception, not the rule, with extensions.
The point is that MediaWiki is *much* more likely to be able
to police and maintain that proper signature management in core code, than in extensions and obscure miscellaneous modules.
If we are just talking about badly maintained and managed code, then how is pushing this code into core going to help? You'll still need to do the effort to maintain it before it improves. For one random unmaintained extension this might work, but there are hundreds. If you make to many of them part of core while not increasing maintenance capacity or without making people care about the code in question, you just end up with the same situation. Except that now the unmaintained code is causing problems for everyone, rather then sitting on a side, not being maintained, which perhaps is because no one cares about the thing any more anyway.
This seems to go into the direction of arguing against having real extensions. I for one, think it is a bad idea to ditch extensions.
The fact that this thought experiment is almost impossible to visualise
without pen and paper is symptomatic of the problem.
Not really. The problem is there now. That understanding of the system is required to pinpoint what is causing the problem is why end users are not flagging these points as being problematic. They just have hassle and know _something_ is going wrong. Understanding all these internal dependencies between components is going to be something non-trivial and something users should not be bothered with in any situation. Lack of understanding is not a symptom of the problem, this lack, at least on the dev side, is the cause of it.
So either the system has to actually *
be* simple (version X of an extension requires version Y of MediaWiki, so if I update my MW install I need to also update extensions (or vice versa)); or it needs to be *properly* abstracted behind a system which makes it *appear* simple (you run some script and everything magically works or gives you informative explanations of why it didn't).
Fully agree.
I attempted to install Wikibase the other day and made a fun discovery.
Installing it properly requires the following (12) extensions
What started this thread was Ryan having problems with installing Wikibase. And I can see why this would not be all that smooth. The components you need is probably not the biggest hassle. After all, you just need to do 4 git clones instead of 3. Then you also need to do a bunch of config, and figure out if you want to use additional functionality.
For instance, you need Scrubuntu if you want to have lua support on the client. As a new user of the software, how will you know you need this or not? Some very good documentation can help, but we don't have this. Putting the lua stuff into its own extension would make this a lot more clear, and would not bother users with the requirement of understanding what this is while they don't even want to use lua. (This model of creating extensions on top of your main extension is something done by SMW and is working very well. Users generally do not get confused by it at all.)
Doing 4 git clones and having some basic understanding of the dependencies is something quite reasonable to expect from developers. As everyone here, including me, is saying, this is not acceptable for users. The problem in case of Wikibase is that it is the only process we currently have, as no one has bothered to do a proper release. You know, one of these things with a version number, release notes, tarballs, etc. If we want this software to be usable by non-devs, this needs to happen. And if it does, it is also going to be useful to devs. So I'm actually quite disappointed we did not have a release yet, and am regularly shouting to people about it.
So Ryan, I agree that currently this is not in good state. I however disagree there is this dichotomy between developers and users, where we can only have it work nicely for one. Attacking the developer process is thus not a good option, as that will just end up hurting everyone, including the end users.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On Sun, Jul 21, 2013 at 9:41 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
What started this thread was Ryan having problems with installing Wikibase. And I can see why this would not be all that smooth. The components you need is probably not the biggest hassle. After all, you just need to do 4 git clones instead of 3. Then you also need to do a bunch of config, and figure out if you want to use additional functionality.
For instance, you need Scrubuntu if you want to have lua support on the client. As a new user of the software, how will you know you need this or not? Some very good documentation can help, but we don't have this. Putting the lua stuff into its own extension would make this a lot more clear, and would not bother users with the requirement of understanding what this is while they don't even want to use lua. (This model of creating extensions on top of your main extension is something done by SMW and is working very well. Users generally do not get confused by it at all.)
Actually, funny enough, I think SMW is one of the more difficult extension sets to use as an end-user. It was the first extension where I saw your Validator code used as a dependency extension. Validator was rejected from being included in core, but now basically every extension you maintain uses it. I didn't bother to say anything about this when it was limited to SMW and your extensions, but now Wikimedia deployed extensions are starting to use them.
Doing 4 git clones and having some basic understanding of the dependencies is something quite reasonable to expect from developers. As everyone here, including me, is saying, this is not acceptable for users. The problem in case of Wikibase is that it is the only process we currently have, as no one has bothered to do a proper release. You know, one of these things with a version number, release notes, tarballs, etc. If we want this software to be usable by non-devs, this needs to happen. And if it does, it is also going to be useful to devs. So I'm actually quite disappointed we did not have a release yet, and am regularly shouting to people about it.
The actual method of getting the extensions isn't that hard. Even if an end-user can't do a git clone, they can have git.wm.o give them a tar or zip. It's specifically the dependencies that are hard. When I attempt to upgrade MediaWiki I currently have to write down all of the extensions, and ensure all of them are compatible with MediaWiki. With some subsets, I need to ensure they are compatible with each other (like SMW, SF, SRF). Now I'm going to need to do that and track the compatibility between extensions and dependency extensions. I'm actually going to have to write an upgrade matrix to upgrade, and that's not OK.
So Ryan, I agree that currently this is not in good state. I however disagree there is this dichotomy between developers and users, where we can only have it work nicely for one. Attacking the developer process is thus not a good option, as that will just end up hurting everyone, including the end users.
I didn't say there's a dichotomy. I believe that we can have both, but this approach isn't it.
- Ryan
Hey,
Validator was rejected from being included in core
I'm happy it did. The code was quite poor at that time (two years back?). And it still is not a hallmark of great design, though certainly not below average MW quality at this point. Regardless of that, I now think it is a bad idea to have it in core and would not petition to have it there in the future.
but now basically every extension you maintain uses it.
Can you please look at objective reality first before making claims about how you would like the world to be so you can bitch about it? Not only are you overly simplifying things, you get simple facts plain wrong.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
Hi,
please, everyone calm down and honesty try harder to assume faith and respect the capabilities of each other. Respect includes to avoid terminology like "to bitch", "to sneak in", "stupid" when describing each other's actions.
One good way is when you are angry about an email, step back, wait a day, calm down, and answer only then. Since this matter is important, but not urgent, there is nothing that requires quick responses. No one's going to make a decision for all of us right away, so there is no need to reply quickly and escalate.
I assume Ryan didn't mean to single out the Wikidata development team. Other teams have done this as well -- the Translate extension depends on ULS, CodeEditor depends on WikiEditor, Semantic Result Formats depends on Semantic MediaWiki etc. I assume Ryan just choose Wikibase because it exemplifies the symptoms so well. Ryan, please correct me if my assumptions are wrong.
The reminder of this mail has two parts, first it tries to explain the motivation of why the Wikidata dev team did things the way we did, and second, it asks this list to please resolve the underlying actual issue.
First:
The original message by Ryan lists the number of dependencies for Wikibase. He lists, e.g. Scribunto as an extension. The question is, how to avoid this dependency? Should we move Scribunto to core? Should we turn Scribunto and Wikibase into one extension?
The same for Babel and ULS, listed as optional extensions.
Another extension mentioned is Diff. Diff provides generic functionality that can be used by different extensions. It seems that in this case the suggestion that Ryan makes is to move it to core. So shall we move functionality that more than one extension depend on generally to core?
One of the reasons for DataValues, Ask and some others being in their own extension is that they already are or are planned very soonish to be reused in SMW. Since it is suggested that generic functionality should be moved to core, would that include functionality shared by Wikibase and SMW? Or how else would we manage that shared code?
Another reason for having separate components like the WikibaseModel or Diff is that they are not MediaWiki extensions, but pure PHP libraries. Any PHP script can reuse them. Since the WikibaseModel is not trivial, this should help with the writing of bots and scripts dealing with Wikidata data. How should we handle such components? Should they be moved to Wikibase and we require every script to depend on the whole of Wikibase, and thus MediaWiki?
If you add everything needed by Wikibase into a single extension, how do you ensure that no unnecessary dependencies creep in? Is there a code-analyzer that can run as part of Jenkins that check that the architecture is not being violated, and that parts of the code to not introduce dependencies on other parts where they should not? Separating such components allows us to check this part of the architecture during CI, which is indeed extremely helpful.
I would indeed be very much interested in better solutions for these questions than we currently have.
As Ryan said in his thread-opening email, "For legitimate library-like extensions I have no constructive alternative, but there must be some sane alternative to this." A lot of the issues would be resolved if we had this constructive alternative. The solution will likely also help to deal with the other dependencies. I hope it is understandable that I do not consider the time of the Wikidata development well spent to replace our architecture with something else, before we have agreed on what this something else should be.
Second:
I would be interested in answers to the above questions. But maybe we really should concentrate on getting the actual question resolved, which has been discussed on this list several times without consensus, those that would allows us to answer the above questions trivially: how should we deal with modularity, extensions, components, etc. in MediaWiki? I hope the answer is not "throw everything in core or into monolithical extensions which do not have dependencies among each other", but let's see what the discussion will bring. Once we have this answer, we can implement the results. Until then I am not sure whether I found it productive to single the Wikidata team out in the way we are doing things.
I do not think the Wikidata development team is anticipating and predetermining answers to those questions, but in order for us to continue develop we had to make some decisions for us. But we don't tell anyone else how to do their job, and we try to stick to current consensus on how to do things. And as soon as we have consensus, we will adopt. We did this previously a few times, and I don't see why we should not do it again.
Cheers, Denny
2013/7/22 Jeroen De Dauw jeroendedauw@gmail.com
Hey,
Validator was rejected from being included in core
I'm happy it did. The code was quite poor at that time (two years back?). And it still is not a hallmark of great design, though certainly not below average MW quality at this point. Regardless of that, I now think it is a bad idea to have it in core and would not petition to have it there in the future.
but now basically every extension you maintain uses it.
Can you please look at objective reality first before making claims about how you would like the world to be so you can bitch about it? Not only are you overly simplifying things, you get simple facts plain wrong.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 -- _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
It seems that a package manager is a reasonable way to manage a growing set of dependencies. Composer seems to me (without knowing too much about it) to be a reasonable package manager.
I've heard the objection expressed that this will require new users of wikibase/etc to "learn composer". Presumably the addition of a composer.json file to the repo won't in itself prevent users from using the released tarballs and ignoring composer altogether, so the only folks who have to work with composer are those developing/admining a mediawiki derivative with hairy dependencies, such that learning composer pays for itself. Correct me if I misunderstand.
Can someone outline (for discussion's sake) the other objections to composer? Otherwise, this discussion does not seem too knotty: someone should submit a patch to "composer-ify" mediawiki/core and the various extensions, the release process should include uploading versions to https://packagist.org/ (optional, since composer can apparently install directly from WMF git), and then people should use it for a while before deciding if something else is needed (which may be just patches to composer). --scott, trying to drain the drama out of this discussion and understand the technical issues.
ps. are patches needed to composer to make this process more lightweight/require less modification to mediawiki/core? I note that mediawiki seems to have support in https://github.com/composer/installersalready.
pps. I think the multiple entry points as a technical issue with the installer? By from my reading, there's no reason why you couldn't write a separate 'mediawiki-composer' shell package that included just stub index.php, api.php, etc files which loaded the appropriate entry point from inside vendor/mediawiki or whatever. So (in my understanding) the changes required to mediawiki/core are minimal. Please correct me if I'm wrong (hopefully with a detailed list of the changes actually required).
On Mon, Jul 22, 2013 at 6:25 AM, Denny Vrandečić < denny.vrandecic@wikimedia.de> wrote:
I assume Ryan didn't mean to single out the Wikidata development team. Other teams have done this as well -- the Translate extension depends on ULS, CodeEditor depends on WikiEditor, Semantic Result Formats depends on Semantic MediaWiki etc. I assume Ryan just choose Wikibase because it exemplifies the symptoms so well. Ryan, please correct me if my assumptions are wrong.
Indeed, this is a general problem, but Wikibase does this quite a bit more than other extensions have in the past. It's been a growing concern of mine as an admin for quite a while now.
The reminder of this mail has two parts, first it tries to explain the motivation of why the Wikidata dev team did things the way we did, and second, it asks this list to please resolve the underlying actual issue.
First:
The original message by Ryan lists the number of dependencies for Wikibase. He lists, e.g. Scribunto as an extension. The question is, how to avoid this dependency? Should we move Scribunto to core? Should we turn Scribunto and Wikibase into one extension?
The same for Babel and ULS, listed as optional extensions.
Another extension mentioned is Diff. Diff provides generic functionality that can be used by different extensions. It seems that in this case the suggestion that Ryan makes is to move it to core. So shall we move functionality that more than one extension depend on generally to core?
One of the reasons for DataValues, Ask and some others being in their own extension is that they already are or are planned very soonish to be reused in SMW. Since it is suggested that generic functionality should be moved to core, would that include functionality shared by Wikibase and SMW? Or how else would we manage that shared code?
This is the hardest scenario of the group. It's likely best that they are in separate extensions. This is still a complication, but is more an issue with MediaWiki's lack of extension management than it is with the modularity of the extensions themselves.
Another reason for having separate components like the WikibaseModel or Diff is that they are not MediaWiki extensions, but pure PHP libraries. Any PHP script can reuse them. Since the WikibaseModel is not trivial, this should help with the writing of bots and scripts dealing with Wikidata data. How should we handle such components? Should they be moved to Wikibase and we require every script to depend on the whole of Wikibase, and thus MediaWiki?
For pure PHP libraries, they could be distributed like pure PHP libraries usually are. They can be packaged for multiple distros and be available via apt/yum/composer (or pear). Having them as MediaWiki extensions is somewhat awkward.
If you add everything needed by Wikibase into a single extension, how do you ensure that no unnecessary dependencies creep in? Is there a code-analyzer that can run as part of Jenkins that check that the architecture is not being violated, and that parts of the code to not introduce dependencies on other parts where they should not? Separating such components allows us to check this part of the architecture during CI, which is indeed extremely helpful.
How does splitting extensions apart make it easier to check this during CI? Is there some automated way to do this when they are split, but not when they are together?
I would indeed be very much interested in better solutions for these questions than we currently have.
As Ryan said in his thread-opening email, "For legitimate library-like extensions I have no constructive alternative, but there must be some sane alternative to this." A lot of the issues would be resolved if we had this constructive alternative. The solution will likely also help to deal with the other dependencies. I hope it is understandable that I do not consider the time of the Wikidata development well spent to replace our architecture with something else, before we have agreed on what this something else should be.
Yes, that's surely a good position to take :).
Second:
I would be interested in answers to the above questions. But maybe we really should concentrate on getting the actual question resolved, which has been discussed on this list several times without consensus, those that would allows us to answer the above questions trivially: how should we deal with modularity, extensions, components, etc. in MediaWiki? I hope the answer is not "throw everything in core or into monolithical extensions which do not have dependencies among each other", but let's see what the discussion will bring. Once we have this answer, we can implement the results. Until then I am not sure whether I found it productive to single the Wikidata team out in the way we are doing things.
My concern was increasing complexity for admins with no effort towards decreasing it. Composer may indeed be an answer to some of the issues. From an admin's perspective using composer shouldn't be much harder than what they are currently expected to do and it would simplify the extension process.
- Ryan
Le 22/07/13 19:31, Ryan Lane a écrit :
Another reason for having separate components like the WikibaseModel or Diff is that they are not MediaWiki extensions, but pure PHP libraries. Any PHP script can reuse them. Since the WikibaseModel is not trivial, this should help with the writing of bots and scripts dealing with Wikidata data. How should we handle such components? Should they be moved to Wikibase and we require every script to depend on the whole of Wikibase, and thus MediaWiki?
For pure PHP libraries, they could be distributed like pure PHP libraries usually are. They can be packaged for multiple distros and be available via apt/yum/composer (or pear). Having them as MediaWiki extensions is somewhat awkward.
Maybe we can move the standalone libraries out of mediawiki/extensions/ to some new namespace such as php/ . Then the standalone libraries are also extensions :-(
<rant> Please no pear. That is a dumb system that needs to disappear. You cant even upload GPL code there, they enforce their own style and require to use PEAR glue code such as status code and autoloader. </rant>
2013/7/22 Ryan Lane rlane32@gmail.com
For pure PHP libraries, they could be distributed like pure PHP libraries usually are. They can be packaged for multiple distros and be available via apt/yum/composer (or pear). Having them as MediaWiki extensions is somewhat awkward.
Yes, agree on that.
How does splitting extensions apart make it easier to check this during CI? Is there some automated way to do this when they are split, but not when they are together?
Yes. Assume you have class Ext which relies on class Core, but class Core should not rely on class Ext, because they are on different architectural levels. If Ext and Core are together, your CI will load them both and you won't notice if you access Ext from Core. No error is thrown. If Core is not together with Ext, then unit-testing Core will not load Ext. A call to Core will fail, and your CI will discover the error. The split helps keeping the architectural integrity of the code, and avoids it being
My concern was increasing complexity for admins with no effort towards decreasing it. Composer may indeed be an answer to some of the issues. From an admin's perspective using composer shouldn't be much harder than what they are currently expected to do and it would simplify the extension process.
Yes, I think so too. Composer could be a viable option, and within Wikidata we are quite happy with it. We would recommend adoption by the wider community. If the wider community chooses another solution for the dependency problem, we will adapt Wikibase to it too.
Cheers, Denny
On Mon, Jul 22, 2013 at 3:23 PM, Denny Vrandečić < denny.vrandecic@wikimedia.de> wrote:
Assume you have class Ext which relies on class Core, but class Core should not rely on class Ext, because they are on different architectural levels. If Ext and Core are together, your CI will load them both and you won't notice if you access Ext from Core. No error is thrown. If Core is not together with Ext, then unit-testing Core will not load Ext. A call to Core will fail, and your CI will discover the error. The split helps keeping the architectural integrity of the code, and avoids it being
Architectural integrity of code is a design-level issue. Continuous integration is a programming and quality assurance-level issue. They have nothing to do with each other, and you can maintain architectural integrity just fine without having to split your singular product into multiple products. If that were true, than the MW core would be split across fifty different repositories. (If Makefiles can compile different parts of a product independently, then so can we.)
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
2013/7/22 Tyler Romeo tylerromeo@gmail.com
Architectural integrity of code is a design-level issue. Continuous integration is a programming and quality assurance-level issue. They have nothing to do with each other, and you can maintain architectural integrity just fine without having to split your singular product into multiple products.
I would disagree with you regarding your statement that architectural integrity and quality assurance have nothing to do with each other. I hope I do not have to explain - but if I do, feel free to ask.
I agree with you regarding your statement that you can maintain architectural integrity without checking it automatically. You can also make sure that code style is not violated manually. Or that code works without unit tests and by testing manually. But considering that reviewing resources are scarce, I prefer to test as much automatically as possibly by CI and relieve the reviewer from considering e.g. the dependency architecture of your classes during a review. I do not see the advantage of doing that manually.
I think that also core would benefit if architectural constraints could be enforced by CI.
If that were true, than the MW core would be split across fifty different repositories. (If Makefiles can compile different parts of a product independently, then so can we.)
I fail to understand what you mean here, sorry.
Le 21/07/13 22:58, Ryan Lane a écrit :
When I attempt to upgrade MediaWiki I currently have to write down all of the extensions, and ensure all of them are compatible with MediaWiki. With some subsets, I need to ensure they are compatible with each other (like SMW, SF, SRF). Now I'm going to need to do that and track the compatibility between extensions and dependency extensions. I'm actually going to have to write an upgrade matrix to upgrade, and that's not OK.
Why would you want to manually keep track of the dependencies when a tool such as composer can handle it for you?
When I attempt to upgrade MediaWiki I currently have to write down all of the extensions, and ensure all of them are compatible with MediaWiki. With some subsets, I need to ensure they are compatible with each other (like SMW, SF, SRF). Now I'm going to need to do that and track the compatibility between extensions and dependency extensions. I'm actually going to have to write an upgrade matrix to upgrade, and that's not OK.
Why would you want to manually keep track of the dependencies when a tool such as composer can handle it for you?
To throw another viewpoint into the mix. If we require composer, we require users to learn to use composer. Some like myself have never used it, and while it’s a skill I should probably learn that will save me considerable time, it may be that not all will find being forced to learn a new piece of software so great.
Of course I could be missing something here.
Thank you, Derric Atzrott
On 07/22/2013 09:20 AM, Derric Atzrott wrote:
If we require composer, we require users to learn to use composer. Some like myself have never used it, and while it’s a skill I should probably learn that will save me considerable time, it may be that not all will find being forced to learn a new piece of software so great.
Of course I could be missing something here.
Composer is becoming widely used in the PHP universe. If your only focus of development is MediaWiki, then, yes, this is a pain.
However, the use of composer could open up whole new possibilities. For the old Perl hands, composer is a lot like cpan. There are a lot of ugly things in there, but it doesn't take too much to find some really useful re-usable components.
And composer would only be a requirement for developers, not end users. There is no reason that we can't still distribute tarballs that a Sysadmin who just maintains a wiki or a wiki farm today would install in the same way he currently does.
Mark.
On Mon, Jul 22, 2013 at 8:36 AM, Mark A. Hershberger mah@everybody.orgwrote:
On 07/22/2013 09:20 AM, Derric Atzrott wrote:
If we require composer, we require users to learn to use composer. Some like myself have never used it, and while it’s a skill I should probably learn that will save me considerable time, it may be that not all will find being forced to learn a new piece of software so great.
Of course I could be missing something here.
Composer is becoming widely used in the PHP universe. If your only focus of development is MediaWiki, then, yes, this is a pain.
However, the use of composer could open up whole new possibilities. For the old Perl hands, composer is a lot like cpan. There are a lot of ugly things in there, but it doesn't take too much to find some really useful re-usable components.
And composer would only be a requirement for developers, not end users. There is no reason that we can't still distribute tarballs that a Sysadmin who just maintains a wiki or a wiki farm today would install in the same way he currently does.
Telling me it's like cpan just brings back awful awful memories...
And a developer, please don't *require* me to use Composer. Don't want it, don't need it.
-Chad
On 07/22/2013 11:43 AM, Chad wrote:
Telling me it's like cpan just brings back awful awful memories...
I apologize. I can't say my experience with cpan was all roses, but it seems that my experience was better than yours. ;)
On Mon, Jul 22, 2013 at 12:00 PM, Mark A. Hershberger mah@everybody.orgwrote:
On 07/22/2013 11:43 AM, Chad wrote:
Telling me it's like cpan just brings back awful awful memories...
I apologize. I can't say my experience with cpan was all roses, but it seems that my experience was better than yours. ;)
The cool kids say "it's like npm" now. --scott
You're not convincing me ;-)
-Chad On Jul 22, 2013 9:06 AM, "C. Scott Ananian" cananian@wikimedia.org wrote:
On Mon, Jul 22, 2013 at 12:00 PM, Mark A. Hershberger <mah@everybody.org
wrote:
On 07/22/2013 11:43 AM, Chad wrote:
Telling me it's like cpan just brings back awful awful memories...
I apologize. I can't say my experience with cpan was all roses, but it seems that my experience was better than yours. ;)
The cool kids say "it's like npm" now. --scott
-- (http://cscott.net) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Just to clear some things up, composer is *not* a package manager. It is actually pretty terrible at being a package manager, mainly because it's not supposed to be one. The purpose of composer is solely dependency management.
Because of that, using it as a package manager requires using some hackish techniques as mentioned above, where you require the MW core as a library (even though it isn't one). That's the main reason I'd consider not using composer.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
On Mon, Jul 22, 2013 at 12:06 PM, C. Scott Ananian cananian@wikimedia.orgwrote:
On Mon, Jul 22, 2013 at 12:00 PM, Mark A. Hershberger <mah@everybody.org
wrote:
On 07/22/2013 11:43 AM, Chad wrote:
Telling me it's like cpan just brings back awful awful memories...
I apologize. I can't say my experience with cpan was all roses, but it seems that my experience was better than yours. ;)
The cool kids say "it's like npm" now. --scott
-- (http://cscott.net) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Mon, Jul 22, 2013 at 12:15 PM, Tyler Romeo tylerromeo@gmail.com wrote:
Just to clear some things up, composer is *not* a package manager. It is actually pretty terrible at being a package manager, mainly because it's not supposed to be one. The purpose of composer is solely dependency management.
Arguably the problem which needs to be solved here is dependency management.
Regardless, the question is: can composer help? It appears that it can: https://www.mediawiki.org/wiki/Composer
I'm interested in having a technical discussion of next steps. What can we do to make this work even better? --scott (who will attempt to ignore future non-technical posts on this thread)
On Mon, Jul 22, 2013 at 12:53 PM, C. Scott Ananian cananian@wikimedia.orgwrote:
Arguably the problem which needs to be solved here is dependency management.
Regardless, the question is: can composer help? It appears that it can: https://www.mediawiki.org/wiki/Composer
I'm interested in having a technical discussion of next steps. What can we do to make this work even better? --scott (who will attempt to ignore future non-technical posts on this thread)
Partially. The issue is that extensions are both packages and dependencies. Scribunto is itself an independent package that a wiki can install, but at the same time it can be a dependency for other extensions. That's where it gets tricky...
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science www.whizkidztech.com | tylerromeo@gmail.com
On Mon, Jul 22, 2013 at 12:55 PM, Tyler Romeo tylerromeo@gmail.com wrote:
Partially. The issue is that extensions are both packages and dependencies. Scribunto is itself an independent package that a wiki can install, but at the same time it can be a dependency for other extensions. That's where it gets tricky...
npm has a newly-added 'peerDependencies' feature. It looks like composer could use a similar feature. Anyone want to work on a composer patch? --scott
Hey,
npm has a newly-added 'peerDependencies' feature. It looks like composer
could use a similar feature. Anyone want to work on a composer patch?
I might be misunderstanding what this peerDependencies does, though if not, then it is different then what is needed for the scenario described below:
Partially. The issue is that extensions are both packages and
dependencies.
Scribunto is itself an independent package that a wiki can install, but at the same time it can be a dependency for other extensions. That's where it gets tricky...
If you want to install Scribuntu, then add it to the requires section of composer.json. If you want the extension that needs scrubuntu, then add the extension to the requires section of composer.json. If you want both, add both. Then run composer install, and you are done. The resolving of dependencies Composer does eliminates trickiness one would run into when doing manual installation.
npm has a newly-added 'peerDependencies' feature. It looks like composer
could use a similar feature. Anyone want to work on a composer patch?
As I understand this peerDependencies thing, it is designed to be able to deal with cycles in your dependency graph. That might be nice to have in such a tool, though I'd argue not the most important feature. If you got cycles in your dependency graph, you got a serious problem with your dependencies, which you should fix. See also: Acyclic Dependencies Principle
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
Hey,
Regardless, the question is: can composer help? It appears that it can:
Yeah, and we've already done most of the steps to make this work for extensions in core. If you got a MW install, you can already install Wikibase (together with all its dependencies), though its going to leave you with a modified composer.json.
The solution proposed by Hashar in some other thread is to have a "MediaWiki installation" package, which just contains a composer.json file with core in it, where people can add dependencies, and then install. Before that will work, we'll need to get rid of "global scope assumptions" in all core files. For instance, WebStart.php has globals in it, accessed without specifying they are globals. This will break when included via Composer, as it will not do so in global scope.
This global scope problem is also present in pretty much all MediaWiki extensions, except for Wikibase and its libraries, where it was fixed in part to not break when used via Composer. This means we will not be able to do things in core and have extensions automatically work. (Unless you do some kind of hack where you promote all variables in the current scope to global scope, though I suspect we do not want to do this :D).
--scott (who will attempt to ignore future non-technical posts on this
thread)
That is a good explicit rule. Thanks, I'll follow it as well.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
On Mon, Jul 22, 2013 at 11:24 AM, Jeroen De Dauw jeroendedauw@gmail.comwrote:
The solution proposed by Hashar in some other thread is to have a "MediaWiki installation" package, which just contains a composer.json file with core in it, where people can add dependencies, and then install. Before that will work, we'll need to get rid of "global scope assumptions" in all core files. For instance, WebStart.php has globals in it, accessed without specifying they are globals. This will break when included via Composer, as it will not do so in global scope.
Easier said than done. I've been fighting this fight for years and we're only about 3 globals closer to doing so. I'm totally in favor of getting rid of global scope usage, but let's please not trivialize the amount of work it will take to get there.
-Chad
Hey,
The solution proposed by Hashar in some other thread is to have a
"MediaWiki installation" package, which just contains a composer.json
file
with core in it, where people can add dependencies, and then install. Before that will work, we'll need to get rid of "global scope
assumptions"
in all core files. For instance, WebStart.php has globals in it, accessed without specifying they are globals. This will break when included via Composer, as it will not do so in global scope.
Easier said than done. I've been fighting this fight for years and we're only about 3 globals closer to doing so. I'm totally in favor of getting rid of global scope usage, but let's please not trivialize the amount of work it will take to get there.
Easier said than done.
Certainly. Else it would probably already be done :)
I've been fighting this fight for years and we're only about 3 globals
closer to doing so.
I suspect you are not understanding me correctly after reading that. I'm talking about referring to global variables as if they where local variables (thus without the "global" keyword) which happens to work if your code is being executed from global scope, but breaks when it is included from any other scope. Getting rid of all globals would be an humongous task and one that is simply not feasible in a short period of time. Luckily we do not need to do this to have core work with Composer.
but let's please not trivialize the amount of work it will take to get
there.
I certainly did not mean to imply doing this is trivial, as I agree with you it is a lot of work. That is why I'm listing this as one of the main obstacles.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 --
The npm peerDependency option is explicitly for extensions/plugins that depend on other extensions. Ie, a jQuery plugin that requires another jQuery plugin to also be loaded (a peer). This is the inter-extension dependency problem that was being discussed, as I understand it. (I could be wrong of course.) --scott On Jul 22, 2013 2:25 PM, "Jeroen De Dauw" jeroendedauw@gmail.com wrote:
Hey,
Regardless, the question is: can composer help? It appears that it can:
Yeah, and we've already done most of the steps to make this work for extensions in core. If you got a MW install, you can already install Wikibase (together with all its dependencies), though its going to leave you with a modified composer.json.
The solution proposed by Hashar in some other thread is to have a "MediaWiki installation" package, which just contains a composer.json file with core in it, where people can add dependencies, and then install. Before that will work, we'll need to get rid of "global scope assumptions" in all core files. For instance, WebStart.php has globals in it, accessed without specifying they are globals. This will break when included via Composer, as it will not do so in global scope.
This global scope problem is also present in pretty much all MediaWiki extensions, except for Wikibase and its libraries, where it was fixed in part to not break when used via Composer. This means we will not be able to do things in core and have extensions automatically work. (Unless you do some kind of hack where you promote all variables in the current scope to global scope, though I suspect we do not want to do this :D).
--scott (who will attempt to ignore future non-technical posts on this
thread)
That is a good explicit rule. Thanks, I'll follow it as well.
Cheers
-- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. ~=[,,_,,]:3 -- _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 22/07/13 15:20, Derric Atzrott a écrit :
To throw another viewpoint into the mix. If we require composer, we require users to learn to use composer. Some like myself have never used it, and while it’s a skill I should probably learn that will save me considerable time, it may be that not all will find being forced to learn a new piece of software so great.
Of course I could be missing something here.
That is indeed a valid argument. Given we migrated our community from subversion to git, I am confident enough that using composer will be very easy to the community. An almost complete tutorial is a few paragraph long, there are ton of resources around and more and more upstream are migrating to composer :-)
So yeah composer adds one more brick, but that solve so many issues that I think it is worth the tiny time investment.
mkdir project; cd project composer require foo/bar require 'vendor/autoload.php';
Done :-]
2013/7/22 Antoine Musso hashar+wmf@free.fr
Given we migrated our community from subversion to git, I am confident enough that using composer will be very easy to the community.
:D
On Fri, Jul 19, 2013 at 2:27 PM, Ryan Lane rlane32@gmail.com wrote:
What if you want to use wikibase 1.23 and MyAbominationExtension 1.5 that requires an incompatible version of DataValues and MyAwesomeExtension 1.0 that requires an incompatible version of ValueView?
You file a bug report against MyAbominationExtension and/or MyAwesomeExtension, telling them they should update their extensions to be compatible with the latest wikibase. Or wait for wikibase 1.24, with updated DataValues and ValueViews. Again, the extension authors are responsible to keep up-to-date.
Or the admin can try to upgrade/downgrade individual components themselves, just like a Fedora developer is free to do so. But if that breaks you get to keep both pieces. --scott
wikitech-l@lists.wikimedia.org