((Jump to the -- TL;DR -- if you just want to answer my question))
Hey guys, right now managing extensions is a complete mess for those of us with a wiki running a dozens of extensions on a VPC. Even if you use git to make things easier. You still need to batch fetch/pull multiple git repos. And then some extensions don't work via git, instead you install them using composer. ((And don't say use mediawiki/extensions.git, that should conflict with composer extensions, doesn't help when an extension uses something like version tags, and appears to have the same issues with bulk updating a few extensions))
I convinced my boss to let me spend some time (when I have no client projects to work on) building a tool to make managing extensions easier for CLI users.
Some notes on the idea here: https://www.mediawiki.org/w/index.php?title=User:Dantman/Code_Ideas&diff...
Please note that this is NOT meant to be the new way we manage extensions forever, this is a hack meant to be used as a workaround to deal with current reality until the grand future when we're supposed to have a complete extension management system built into core with its own web interface.
-- TL;DR --
Now, my question. Do you guys want me to build the local tool in PHP as a maintenance script or in node.js?
Actually perhaps I should instead ask "Are you guys fine if I build this in node.js?" because I have a feeling this will be hell to develop if I have to write it in PHP.
PHP Pros: - We could bundle this with MediaWiki core if people like it. - You don't need to install Node.js (though it's not 'that' hard).
PHP Cons: - Every time I've tried using PHP proc functions I've had to spend endless time debugging. And this tool requires dozens of proc calls. - The tool is going to be difficult to access until at least the next MediaWiki version, since it won't be bundled. -- It may end up useless for a bunch of people right now when it's supposed to help. -- It may also end up locked down so only users with more recent installations of MediaWiki may use it (
Node.js Pros: - Once you have node, getting the tool will be as trivial as `[sudo] npm install -g mediawiki-...something...`. - The tool will be available for and should work on any MediaWiki version you can get a current extension to work on, not just future releases. - Executing git and composer from node to download things is trivial. -- I not only expect it to be easy, I'm already doing it. The server code is Node.js and is already happily chugging away fetching all our gerrit based git repos, doing multiple in parallel. - I could make parts of the tool interactive and much more user friendly.
I say working in node is sure better than not working at all. :D
Keep it up!
-- brion
On Thu, Feb 12, 2015 at 3:45 PM, Daniel Friesen daniel@nadir-seen-fire.com wrote:
((Jump to the -- TL;DR -- if you just want to answer my question))
Hey guys, right now managing extensions is a complete mess for those of us with a wiki running a dozens of extensions on a VPC. Even if you use git to make things easier. You still need to batch fetch/pull multiple git repos. And then some extensions don't work via git, instead you install them using composer. ((And don't say use mediawiki/extensions.git, that should conflict with composer extensions, doesn't help when an extension uses something like version tags, and appears to have the same issues with bulk updating a few extensions))
I convinced my boss to let me spend some time (when I have no client projects to work on) building a tool to make managing extensions easier for CLI users.
Some notes on the idea here:
https://www.mediawiki.org/w/index.php?title=User:Dantman/Code_Ideas&diff...
Please note that this is NOT meant to be the new way we manage extensions forever, this is a hack meant to be used as a workaround to deal with current reality until the grand future when we're supposed to have a complete extension management system built into core with its own web interface.
-- TL;DR --
Now, my question. Do you guys want me to build the local tool in PHP as a maintenance script or in node.js?
Actually perhaps I should instead ask "Are you guys fine if I build this in node.js?" because I have a feeling this will be hell to develop if I have to write it in PHP.
PHP Pros:
- We could bundle this with MediaWiki core if people like it.
- You don't need to install Node.js (though it's not 'that' hard).
PHP Cons:
- Every time I've tried using PHP proc functions I've had to spend
endless time debugging. And this tool requires dozens of proc calls.
- The tool is going to be difficult to access until at least the next
MediaWiki version, since it won't be bundled. -- It may end up useless for a bunch of people right now when it's supposed to help. -- It may also end up locked down so only users with more recent installations of MediaWiki may use it (
Node.js Pros:
- Once you have node, getting the tool will be as trivial as `[sudo] npm
install -g mediawiki-...something...`.
- The tool will be available for and should work on any MediaWiki
version you can get a current extension to work on, not just future releases.
- Executing git and composer from node to download things is trivial.
-- I not only expect it to be easy, I'm already doing it. The server code is Node.js and is already happily chugging away fetching all our gerrit based git repos, doing multiple in parallel.
- I could make parts of the tool interactive and much more user friendly.
-- ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This seems to have significant overlap with https://www.mediawiki.org/wiki/Requests_for_comment/Improving_extension_mana... .
Personally I think we should standardize around one tool and Composer seems the best positioned for that. So instead of maintaining N competing solutions with different strengths and weaknesses and leave the user to ponder which one to use, I think it would make more sense to focus on the holes in the Composer workflow and write small tools to plug them (e.g. a web interface to use Composer on servers without a shell access, or a way to disable/re-enable extensions without fully uninstalling them).
In that spirit, maybe writing a composer.json (or composer.lock) generator could be a good way of making the extension + git branch selection user friendly? Once you know which version of which extension to install, Composer should be able to do the job. The generator could even be a web service with a nice GUI, like the short url builder (which rocks by the way, thanks for maintaining it!)
On 2015-02-14 1:23 PM, Gergo Tisza wrote:
This seems to have significant overlap with https://www.mediawiki.org/wiki/Requests_for_comment/Improving_extension_mana... .
As I mentioned this tool is a workaround to deal with current reality. Not the "grand way forward". Whenever that happens, the tool will disappear.
The goal here is to make it so that whenever I (or someone else) has to mange the extensions of an extension (setting up a staging wiki for a client project, upgrading the extensions on one of the wiki I host, or helping reprap.org upgrade) today, the job is easier to do until we have a nice web interface for it.
Personally I think we should standardize around one tool and Composer seems the best positioned for that. So instead of maintaining N competing solutions with different strengths and weaknesses and leave the user to ponder which one to use, I think it would make more sense to focus on the holes in the Composer workflow and write small tools to plug them (e.g. a web interface to use Composer on servers without a shell access, or a way to disable/re-enable extensions without fully uninstalling them).
I don't believe that composer is the grand way forward. -- TL;DR -- I do believe that composer is a great way to use 3rd party extensions. However I do not believe it is the way forward in extension installation.
Extensions/plugins are not libraries. Frankly I want to give whoever built the addition to Composer that allows packages to install as WordPress plugins and MediaWiki extensions a good kick.
Besides other things like all our extensions being hosted by a third party and mixed in with libraries where they don't belong. Composer doesn't even expose all the data we need to do a PROPER extension management interface.
i18n-ed name and description, extension type, and MediaWiki.org docs primarily, but there will likely be more. -- / TL;DR --
In that spirit, maybe writing a composer.json (or composer.lock) generator could be a good way of making the extension + git branch selection user friendly? Once you know which version of which extension to install, Composer should be able to do the job. The generator could even be a web service with a nice GUI, like the short url builder (which rocks by the way, thanks for maintaining it!)
As I don't believe composer is the right way forward, building something to try and get everyone to use it would be counter-productive.
Anyways we likely wouldn't get all extensions to start using it. We have over 600 extensions. And not everyone even buys into composer. I'd still have the same problem.
Even if it did work I'd end up creating new problems for myself. Now every time I want to install 'ParserFunctions' I'd have to look up what the composer name for it is.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
Hi,
Extensions/plugins are not libraries. Frankly I want to give whoever built the addition to Composer that allows packages to install as WordPress plugins and MediaWiki extensions a good kick.
Thanks for your elaborated inside but somehow this doesn't tell the whole story.
Take for example our extension (SMW but others are welcome as well) which depends on libraries and other extensions as well. I don't think it is practical nor good practice to have a user to write down each dependency and start kicking the git downland for each of them just to figure out at some point that s(he) missed something or caught the wrong hash tag and suddenly finds him/herself in a dependency nightmare.
Knowing that I can specify what version works with which library and have tool that lets me forget about any of the dependency is what I want, I don't need a WMF special tool that may or may not work in future.
I don't want a tool that increase technical debt on my end and using Composer (which is supported by more developers than WMF can hire) creates a certain confidence that the tool does what it is claimed for namely a tool to resolve dependencies.
Having Composer resolving dependencies and at the same to put it into the right place (thanks for the `mediawiki-extension` type) together with a PSR-4 autoloader is the icing on the cake to make our extension deployable.
If your extension does not depend on anything then this is good but not all extension are going for the golden goal of putting code in one line (or for that matter in one include folder).
Some extensions try to isolate code by its responsibility and do make an effort to separate those into different components, arguing that only MW can strive towards "Librarization" is a bit short-sighted.
Cheers
On 2/15/15, Daniel Friesen daniel@nadir-seen-fire.com wrote:
On 2015-02-14 1:23 PM, Gergo Tisza wrote:
This seems to have significant overlap with https://www.mediawiki.org/wiki/Requests_for_comment/Improving_extension_mana... .
As I mentioned this tool is a workaround to deal with current reality. Not the "grand way forward". Whenever that happens, the tool will disappear.
The goal here is to make it so that whenever I (or someone else) has to mange the extensions of an extension (setting up a staging wiki for a client project, upgrading the extensions on one of the wiki I host, or helping reprap.org upgrade) today, the job is easier to do until we have a nice web interface for it.
Personally I think we should standardize around one tool and Composer seems the best positioned for that. So instead of maintaining N competing solutions with different strengths and weaknesses and leave the user to ponder which one to use, I think it would make more sense to focus on the holes in the Composer workflow and write small tools to plug them (e.g. a web interface to use Composer on servers without a shell access, or a way to disable/re-enable extensions without fully uninstalling them).
I don't believe that composer is the grand way forward. -- TL;DR -- I do believe that composer is a great way to use 3rd party extensions. However I do not believe it is the way forward in extension installation.
Extensions/plugins are not libraries. Frankly I want to give whoever built the addition to Composer that allows packages to install as WordPress plugins and MediaWiki extensions a good kick.
Besides other things like all our extensions being hosted by a third party and mixed in with libraries where they don't belong. Composer doesn't even expose all the data we need to do a PROPER extension management interface.
i18n-ed name and description, extension type, and MediaWiki.org docs primarily, but there will likely be more. -- / TL;DR --
In that spirit, maybe writing a composer.json (or composer.lock) generator could be a good way of making the extension + git branch selection user friendly? Once you know which version of which extension to install, Composer should be able to do the job. The generator could even be a web service with a nice GUI, like the short url builder (which rocks by the way, thanks for maintaining it!)
As I don't believe composer is the right way forward, building something to try and get everyone to use it would be counter-productive.
Anyways we likely wouldn't get all extensions to start using it. We have over 600 extensions. And not everyone even buys into composer. I'd still have the same problem.
Even if it did work I'd end up creating new problems for myself. Now every time I want to install 'ParserFunctions' I'd have to look up what the composer name for it is.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 2015-02-14 11:30 PM, James HK wrote:
Hi,
Extensions/plugins are not libraries. Frankly I want to give whoever built the addition to Composer that allows packages to install as WordPress plugins and MediaWiki extensions a good kick.
Thanks for your elaborated inside but somehow this doesn't tell the whole story.
Take for example our extension (SMW but others are welcome as well) which depends on libraries and other extensions as well. I don't think it is practical nor good practice to have a user to write down each dependency and start kicking the git downland for each of them just to figure out at some point that s(he) missed something or caught the wrong hash tag and suddenly finds him/herself in a dependency nightmare.
Knowing that I can specify what version works with which library and have tool that lets me forget about any of the dependency is what I want, I don't need a WMF special tool that may or may not work in future.
I don't want a tool that increase technical debt on my end and using Composer (which is supported by more developers than WMF can hire) creates a certain confidence that the tool does what it is claimed for namely a tool to resolve dependencies.
Having Composer resolving dependencies and at the same to put it into the right place (thanks for the `mediawiki-extension` type) together with a PSR-4 autoloader is the icing on the cake to make our extension deployable.
If your extension does not depend on anything then this is good but not all extension are going for the golden goal of putting code in one line (or for that matter in one include folder).
Some extensions try to isolate code by its responsibility and do make an effort to separate those into different components, arguing that only MW can strive towards "Librarization" is a bit short-sighted.
Cheers
git vs. composer is a false dichotomy.
I don't believe that using git and ExtensionDistributor tarballs forever is the correct way forward. But I also do not believe shoving all our extensions into composer is the correct way to fix that either.
Composer doesn't fix all the issues and doesn't expose data needed to build a proper system. And it creates release restrictions (do real releases [maintain an explicit version, follow semver] and tag them), which not every extension needs and many authors don't want to deal with, which precludes its use as the primary extension installation system.
---- We need a proper interface and system for Extension management (and by extension, configuration management).
Some notes: - It needs to support the requirements of everything from Semantic MediaWiki to a lowly extension like Poem. - The interface must not be English only, extension names/descriptions for uninstalled extensions should be translated for every extension that will be translated on Special:Version when the extension is installed. - Extensions should be categorized in the same way they are on Special:Version. - A search interface is necessary for installing new extensions. - It should automatically install composer libraries (even if the extension itself isn't published) automatically. Whether this is done by integration of composer/composer or packaging libraries automatically on the origin side. - It should understand when it needs to take installation steps like upgrading the database. ----
It would also be short sited to believe that a system built as a simple wrapper around composer could solve all the version related issues we have. Besides dep-hell, extensions also have incompatibilities with MediaWiki releases themselves.
If I install mediawiki/semantic-media-wiki@2.0.1 today with composer, it will not tell me if it supports MediaWiki 1.16 nor will it tell me if it supports MediaWiki 1.26.
Practically no extension author bothers testing on each release how far back their extension works. Or bother making a note of that which composer or another program could read. And it's impossible for them to know at the time of publishing a release if a new release of MediaWiki will work with their extension or not.
The improving extension management RFC is a step forward in one part of this (a proper way to declare compatibility).
However a proper system would also involve running tests for extensions and accepting reports of success or failure with a version of an extension from users.
All 3 sources of data would be exposed in the same API a MediaWiki install's extension management page uses to search and query data about extensions.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
Hi,
I certainly don't want to drag this discussion but please let me clarify:
If I install mediawiki/semantic-media-wiki@2.0.1 today with composer, it will not tell me if it supports MediaWiki 1.16 nor will it tell me if it supports MediaWiki 1.26.
It is possible but the MediaWikiVersionFetcher in ComposerHookHandler is only loaded by Composer starting with MW 1.25 [0] allowing for such check to take place but since we need to support MW 1.19 we cannot not leverage on that.
Practically no extension author bothers testing on each release how far back their extension works. Or bother making a note of that which composer or another program could read. And it's impossible for them to know at the time of publishing a release if a new release of MediaWiki will work with their extension or not.
We currently have ~72% coverage which makes testing a crucial part to ensure that previous and upcoming releases can be supported (for 1.25 we already run into some failing integration tests allowing for a prediction of a potential issue) other extensions of the same family have an even higher coverage rate which helps us to prognosticate whether a tested MW version can be supported or not, and of course such effort is not feasible for every extension.
extension itself isn't published) automatically. Whether this is done by integration of composer/composer or packaging libraries automatically on the origin side.
- It should understand when it needs to take installation steps like
upgrading the database.
Yes, we are in agreement that an interface (including search, status update etc.) is necessary to make the user experience more pleasant for one or a bundle of extensions. I just don't want to see that dependency management suddenly becomes a trifle where the management of dependencies in itself is reinvented.
[0] https://github.com/wikimedia/mediawiki/blob/master/composer.json#L45
Cheers
On 2/15/15, Daniel Friesen daniel@nadir-seen-fire.com wrote:
On 2015-02-14 11:30 PM, James HK wrote:
Hi,
Extensions/plugins are not libraries. Frankly I want to give whoever built the addition to Composer that allows packages to install as WordPress plugins and MediaWiki extensions a good kick.
Thanks for your elaborated inside but somehow this doesn't tell the whole story.
Take for example our extension (SMW but others are welcome as well) which depends on libraries and other extensions as well. I don't think it is practical nor good practice to have a user to write down each dependency and start kicking the git downland for each of them just to figure out at some point that s(he) missed something or caught the wrong hash tag and suddenly finds him/herself in a dependency nightmare.
Knowing that I can specify what version works with which library and have tool that lets me forget about any of the dependency is what I want, I don't need a WMF special tool that may or may not work in future.
I don't want a tool that increase technical debt on my end and using Composer (which is supported by more developers than WMF can hire) creates a certain confidence that the tool does what it is claimed for namely a tool to resolve dependencies.
Having Composer resolving dependencies and at the same to put it into the right place (thanks for the `mediawiki-extension` type) together with a PSR-4 autoloader is the icing on the cake to make our extension deployable.
If your extension does not depend on anything then this is good but not all extension are going for the golden goal of putting code in one line (or for that matter in one include folder).
Some extensions try to isolate code by its responsibility and do make an effort to separate those into different components, arguing that only MW can strive towards "Librarization" is a bit short-sighted.
Cheers
git vs. composer is a false dichotomy.
I don't believe that using git and ExtensionDistributor tarballs forever is the correct way forward. But I also do not believe shoving all our extensions into composer is the correct way to fix that either.
Composer doesn't fix all the issues and doesn't expose data needed to build a proper system. And it creates release restrictions (do real releases [maintain an explicit version, follow semver] and tag them), which not every extension needs and many authors don't want to deal with, which precludes its use as the primary extension installation system.
We need a proper interface and system for Extension management (and by extension, configuration management).
Some notes:
- It needs to support the requirements of everything from Semantic
MediaWiki to a lowly extension like Poem.
- The interface must not be English only, extension names/descriptions
for uninstalled extensions should be translated for every extension that will be translated on Special:Version when the extension is installed.
- Extensions should be categorized in the same way they are on
Special:Version.
- A search interface is necessary for installing new extensions.
- It should automatically install composer libraries (even if the
extension itself isn't published) automatically. Whether this is done by integration of composer/composer or packaging libraries automatically on the origin side.
- It should understand when it needs to take installation steps like
upgrading the database.
It would also be short sited to believe that a system built as a simple wrapper around composer could solve all the version related issues we have. Besides dep-hell, extensions also have incompatibilities with MediaWiki releases themselves.
If I install mediawiki/semantic-media-wiki@2.0.1 today with composer, it will not tell me if it supports MediaWiki 1.16 nor will it tell me if it supports MediaWiki 1.26.
Practically no extension author bothers testing on each release how far back their extension works. Or bother making a note of that which composer or another program could read. And it's impossible for them to know at the time of publishing a release if a new release of MediaWiki will work with their extension or not.
The improving extension management RFC is a step forward in one part of this (a proper way to declare compatibility).
However a proper system would also involve running tests for extensions and accepting reports of success or failure with a version of an extension from users.
All 3 sources of data would be exposed in the same API a MediaWiki install's extension management page uses to search and query data about extensions.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 2015-02-15 2:05 AM, James HK wrote:
Hi,
I certainly don't want to drag this discussion but please let me clarify:
If I install mediawiki/semantic-media-wiki@2.0.1 today with composer, it will not tell me if it supports MediaWiki 1.16 nor will it tell me if it supports MediaWiki 1.26.
It is possible but the MediaWikiVersionFetcher in ComposerHookHandler is only loaded by Composer starting with MW 1.25 [0] allowing for such check to take place but since we need to support MW 1.19 we cannot not leverage on that.
When we introduced composer isn't really relevant, since I'm talking about the lack of information in relation to an extension's compatibility with MediaWiki in composer. Something that grows as we have more releases with composer support.
A hypothetical future example might be better than picking an old release number and one that doesn't exist.
We've had composer support for ages, the stable release is MW 1.42, there is a supported MW 1.40 LTS and a MW 1.36 LTS, and MW 1.43 is in development. Composer doesn't contain the information necessary to say: - If the latest version of mediawiki/semantic-media-wiki, 4.2.2, is compatible with the recent MW 1.40 LTS, or what the most recent release is compatible. - Same question, but for the MW 1.44 release that doesn't exist yet. Since the composer requires cannot be updated when MW 1.44 is released to say that 4.2.2 isn't compatible with it.
Ok maybe I should adjust that, since you could put a `mediawiki/mediawiki >= 1.39` by that point.
There are two problems: A) Even if you guys can put `mediawiki/mediawiki >= 1.39` in mediawiki/semantic-media-wiki. The average extension author will not bother to maintain a copy of the supported MW 1.44, MW 1.40 LTS, and MW 1.36 LTS to test their simple extension in, much less bother to find out that it supports `mediawiki/mediawiki >= 1.32`. B) It's impossible for you guys to know when you release 4.2.2, that MW 1.44 will introduce a breaking change that breaks 4.2.2 and you need to declare `mediawiki/mediawiki >= 1.39, < 1.44`.
The relationship between apps/libraries that use libraries, where semver works, is different from the relationship between Extensions and MediaWiki core.
In an app/library there's a fair chance that a new `2.0.0` of a library will break, so it's fine to depend on `^1.3.9` and wait till it's released to make any code changes you need to change to `^2.0.0`.
But for an Extension. MediaWiki is huge with piles of different components. Each release is almost guaranteed to include a breaking change. However most of these don't affect you and as long as you're not using a part that's currently getting a lot of attention for vast improvement (like skinning and extension management is now), your extension will likely work the exact same for the next half dozen releases (Sudo seems to have went at least from 2011-12 to 2013-10 without needing changes to keep compatibility with new releases). So you can't really declare that your extension can't use the next MW release when you don't know if it will or won't. Since then users wouldn't be able to install it as what MW release a user has installed is completely out of your control.
extension itself isn't published) automatically. Whether this is done by integration of composer/composer or packaging libraries automatically on the origin side.
- It should understand when it needs to take installation steps like
upgrading the database.
Yes, we are in agreement that an interface (including search, status update etc.) is necessary to make the user experience more pleasant for one or a bundle of extensions. I just don't want to see that dependency management suddenly becomes a trifle where the management of dependencies in itself is reinvented.
Ok, then lets break it down into smaller pieces of what is good and bad about using composer to manage extensions.
Some of the "bad" things I can think of about the composer direction: - Being forced to use composer names "author/name" when we already have a standard. - Being stuck using composer.json when extension.json is better matched with MediaWiki (things like i18n support for name/description, the same names we've been using forever and use in the extensions themselves). - Extensions being distributed and 100% dependent on a third party instead of distributed through MediaWiki.org. - Lack of support for alternatives to maintaining semver releases for smaller extensions. Such as just making every new batch of commits available to install. Or just maintaining a few REL#_## branches. (The latter two essentially how ExtensionDistributor works, but could still be nicer methods with more polish).
I guess there is merit to wanting to avoid reinventing (or reimplementing) dependency management. But likewise I don't want to shoehorn the entire MediaWiki extension ecosystem into the Composer library ecosystem to do so.
I wonder how accessible composer's dependency management code is. That is, whether it's possible to depend on composer/composer and use parts of its code not just for installing composer libraries, but also to implement dependency management in extension.json.
----
Btw, side thought. I wonder how the requirement to bump version numbers and make releases clashes with all the regular TWN message updates extensions get.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
wikitech-l@lists.wikimedia.org