My logging changes [0][1][2][3] are getting closer to being mergeable (the first has already been merged). Tony Thomas' Swift Mailer change [4] is also progressing. Both sets of changes introduce the concept of specifying external library dependencies, both required and suggested, to mediawiki/core.git via composer.json. Composer can be used by people directly consuming the git repository to install and manage these dependencies. I gave a example set of usage instructions in the commit message for my patch that introduced the dependency on PSR-3 [0]. In the production cluster, on Jenkins job runners and in the tarball releases we will want a different solution.
My idea of how to deal with this is to create a new gerrit repository (mediawiki/core/vendor.git?) that contains a composer.json file similar to the one I had in patch set 7 of my first logging patch [5]. This composer.json file would be used to tell Composer the exact versions of libraries to download. Someone would manually run Composer in a checkout of this repository and then commit the downloaded content, composer.lock file and generated autoloader.php to the repository for review. We would then be able to branch and use this repository as git submodule in the wmf/1.2XwmfY branches that are deployed to production and ensure that it is checked out along with mw-core on the Jenkins nodes. By placing this submodule at $IP/vendor in mw-core we would be mimicking the configuration that direct users of Composer will experience. WebStart.php already includes $IP/vendor/autoload.php when present so integration with the rest of wm-core should follow from that.
It would also be possible to add this repo to the tarballs for distribution. There will probably need to be some adjustments for that process however and the final result may be that release branches update the mediawiki/core composer.json and provide a composer.lock along with a pre-populated vendor directory. I would be glad to participate in discussions of that use case, but we will have about 6 months before we need to solve it (and a new release management RFC to resolve between now and then).
There are several use cases to consider for the general solution:
== Adding/updating a library == * Update composer.json in mediawiki/core/vendor.git * Run `composer update` locally to download library (and dependencies) * Run `composer dump-autoload --optimize` to make an optimized autoloader.php * Commit changes * Push changes for review in gerrit
== Hotfix for an external library == At some point we will run into a bug or missing feature in a Composer managed library that we need to work around with a patch. Obviously we will attempt to upstream any such fixes (otherwise what's the point of this whole exercise?). To keep from blocking things for our production cluster we would want to fork the upstream, add our patch for local use and upstream the patch. During the time that the patch was pending review in the upstream we would want to use our locally patched version in production and Jenkins.
Composer provides a solution for this with its "repository" package source. The Composer documentation actually gives this exact example in their discussion of the "vcs" repository type [6]. We would create a gerrit repository tracking the external library, add our patch(es), adjust the composer.json file in mediawiki/core/vendor.git to reference our fork, and finally run Composer in mediawiki/core/vendor.git to pull in our patched version.
== Adding a locally developed library == The Platform Core team has been talking about extracting libraries from mw-core and/or extensions to be published externally. This may be done for any and all of the current $IP/includes/libs classes and possibly other content from core such as FormatJson.
My idea for this would be to create a new gerrit repository for each exported project. The project repo would contain a composer.json manifest describing the project correctly to be published at packagist.org like most Composer installable libraries. In the mediawiki/core/vendor.git composer.json file we would pull these libraries just like any third-party developed library. This isn't functionally much different than the way that we use git submodules today. There is one extra level of indirection when a library is changed. The mediawiki/core/vendor.git will have to be updated with the new library version before the hash for the git submodule of mediawiki/core/vendor.git is updated in a deploy or release branch.
== wmf/1.XwmfY branches == The make-wmf-branch script (found in mediawiki/tools/release.git) is used to create the weekly release branches that are deployed by the "train" on each Thursday. This script would be updated to branch the new mediawiki/core/vendor.git repository and add the version appropriate branch as a submodule of mediawiki/core.git on the wmf/* branch. This is functionally exactly what we do for extensions today.
== Updating a deployment branch == SWAT deploys often deploy bug fixes for extensions and core that can't wait for the next train release. It is a near certainty that mediawiki/core/vendor.git will have the same need. The process for updating mediawiki/core/vendor.git will be almost the same as updating an extension.
* Follow the adding/updating library or hotfix instructions to get the changes merged into the mediawiki/core/vendor.git master branch. * Cherry-pick the change into the proper deployment branch * Merge the cherry-pick * Update the git submodule for mediawiki/core/vendor.git in the appropriate deployed branch * Pull update to tin * sync-dir to deploy to cluster
== Security fixes == This is a special case of upstreaming a patch. A security patch would be applied directly on the deployed branch of mediawiki/core/vendor.git as we would do for any extension. The vulnerability and patch must then be submitted upstream in a responsible manner and tracked for resolution.
== Jenkins == The Jenkins jobs that checkout and run tests involving mediawiki/core would need to be amended to also checkout the mediawiki/core/vendor.git in the appropriate location before running tests.
What use cases did I miss? What other concerns do we have for this process?
[0]: https://gerrit.wikimedia.org/r/#/c/119939/ [1]: https://gerrit.wikimedia.org/r/#/c/119940/ [2]: https://gerrit.wikimedia.org/r/#/c/119941/ [3]: https://gerrit.wikimedia.org/r/#/c/119942/ [4]: https://gerrit.wikimedia.org/r/#/c/135290/ [5]: https://gerrit.wikimedia.org/r/#/c/119939/7/libs/composer.json,unified [6]: https://getcomposer.org/doc/05-repositories.md#vcs
Bryan
2014-05-29 20:27 GMT+03:00 Bryan Davis bd808@wikimedia.org:
What use cases did I miss? What other concerns do we have for this process?
The email subject does not cover third party users, so apologies if this is not the correct place for this.
Currently updating translatewiki.net codebase is annoying, as git does not handle files replaced with symlinks well.
I am not happy since I had to spent unplanned effort to be able install extensions via composer just a short while ago. We replace composer.json with symlink to our configuration repo, where it includes the extension we install via composer.
Ori has come up with a good solution for this issue, but that solution requires changes to Composer. To me it looks like nobody is currently working on that.
Is there a way I can expedite this fix?
Thank you for your efforts to bring composer support to MediaWiki. I wish the process was less rough for me.
-Niklas
On Thu, May 29, 2014 at 2:38 PM, Niklas Laxström niklas.laxstrom@gmail.com wrote:
2014-05-29 20:27 GMT+03:00 Bryan Davis bd808@wikimedia.org:
What use cases did I miss? What other concerns do we have for this process?
The email subject does not cover third party users, so apologies if this is not the correct place for this.
We need to hash out the third-party issues somewhere. I suppose this thread is as good a place as any to bring up your issues.
Currently updating translatewiki.net codebase is annoying, as git does not handle files replaced with symlinks well.
I am not happy since I had to spent unplanned effort to be able install extensions via composer just a short while ago. We replace composer.json with symlink to our configuration repo, where it includes the extension we install via composer.
Ori has come up with a good solution for this issue, but that solution requires changes to Composer. To me it looks like nobody is currently working on that.
Is there a way I can expedite this fix?
I think bug 65188 [0] is the solution suggested by Ori that you are referring to. Would this alone be enough to fix the problems for translatewiki.net? More directly, is translatewiki.net using the top level composer.json file to manage anything other than extensions? In the near term is there any better work around for you (and others in a similar position) other than running `git update-index --assume-unchanged composer.json` in your local checkout to make it ignore anything that happens to composer.json?
Thank you for your efforts to bring composer support to MediaWiki. I wish the process was less rough for me.
Thanks for the thanks. :) Sorry for the pain, but you are one of the power users who helps us work out the kinks.
[0]: https://bugzilla.wikimedia.org/show_bug.cgi?id=65188
Bryan
2014-05-30 0:57 GMT+03:00 Bryan Davis bd808@wikimedia.org:
I think bug 65188 [0] is the solution suggested by Ori that you are referring to. Would this alone be enough to fix the problems for translatewiki.net? More directly, is translatewiki.net using the top level composer.json file to manage anything other than extensions? In the near term is there any better work around for you (and others in a similar position) other than running `git update-index --assume-unchanged composer.json` in your local checkout to make it ignore anything that happens to composer.json?
Now that composer.json also includes dependencies for core, ignoring changes to it would also break things.
To make things worse, I noticed on my development environment that our own scap-equivalent will just go on to run composer update even if the file conflicted. This causes it to remove the extensions and libraries we currently install via composer, also breaking the site.
-Niklas
To make things worse, I noticed on my development environment that our own scap-equivalent will just go on to run composer update even if the file conflicted. This causes it to remove the extensions and libraries we currently install via composer, also breaking the site.
I hope for the sake of all-non WMF users that already use Composer to install extensions that the proposed changes are not making things worse (well it doesn't seem so).
Cheers
On 6/2/14, Niklas Laxström niklas.laxstrom@gmail.com wrote:
2014-05-30 0:57 GMT+03:00 Bryan Davis bd808@wikimedia.org:
I think bug 65188 [0] is the solution suggested by Ori that you are referring to. Would this alone be enough to fix the problems for translatewiki.net? More directly, is translatewiki.net using the top level composer.json file to manage anything other than extensions? In the near term is there any better work around for you (and others in a similar position) other than running `git update-index --assume-unchanged composer.json` in your local checkout to make it ignore anything that happens to composer.json?
Now that composer.json also includes dependencies for core, ignoring changes to it would also break things.
To make things worse, I noticed on my development environment that our own scap-equivalent will just go on to run composer update even if the file conflicted. This causes it to remove the extensions and libraries we currently install via composer, also breaking the site.
-Niklas
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hey,
To make things worse, I noticed on my development environment that our
own scap-equivalent will just go on to run composer update even if the file conflicted. This causes it to remove the extensions and libraries we currently install via composer, also breaking the site.
I hope for the sake of all-non WMF users that already use Composer to install extensions that the proposed changes are not making things worse (well it doesn't seem so).
I would also like to express my disappointment at third party users being thrown under the bus once again. Several people have been putting a lot of effort into supporting third party users, so it really saddens me that this is dismissed as an irrelevance by some so easily.
Cheers
-- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate Evil software architect at Wikimedia Germany ~=[,,_,,]:3
I would also like to express my disappointment at third party users being thrown under the bus once again. Several people have been putting a lot of effort into supporting third party users, so it really saddens me that this is dismissed as an irrelevance by some so easily. Third party users were not thrown under the bus. Unfortunately, the solution you are looking for in terms of extension installation is not yet available with the current tools available. That is just the unfortunate truth. We are not going to misuse libraries and hack together MediaWiki just so extension installation can be *slightly* easier.
-- Tyler Romeo 0xC86B42DF
That is just the unfortunate truth. We are not going to misuse libraries and hack together MediaWiki just so extension installation can be *slightly* easier.
This sort of behaviour towards non-WMF extension developers is interesting and if your objective is to alienate (as with the attitude above) volunteer developers then your on the right path.
Some people forget that users not always choose MW because of its software but because it provides some extensions that people outside of the WMF cluster find useful.
On 6/3/14, Tyler Romeo tylerromeo@gmail.com wrote:
I would also like to express my disappointment at third party users being thrown under the bus once again. Several people have been putting a lot of effort into supporting third party users, so it really saddens me that this is dismissed as an irrelevance by some so easily. Third party users were not thrown under the bus. Unfortunately, the solution you are looking for in terms of extension installation is not yet available with the current tools available. That is just the unfortunate truth. We are not going to misuse libraries and hack together MediaWiki just so extension installation can be *slightly* easier.
-- Tyler Romeo 0xC86B42DF _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This sort of behaviour towards non-WMF extension developers is interesting and if your objective is to alienate (as with the attitude above) volunteer developers then your on the right path.
Some people forget that users not always choose MW because of its software but because it provides some extensions that people outside of the WMF cluster find useful. Considering I *am* a non-WMF extension developer I don’t see how your argument is relevant.
And as I literally just said in my previous, the goal was not to disadvantage third-party extension developers. Composer is simply not meant to be used in the way it was shoe-horned into MediaWiki. I’m not going to re-explain this every time because it is in multiple places on Bugzilla and in this mailing list.
-- Tyler Romeo 0xC86B42DF
On Mon, Jun 2, 2014 at 6:37 PM, James HK jamesin.hongkong.1@gmail.com wrote:
That is just the unfortunate truth. We are not going to misuse libraries and hack together MediaWiki just so
extension
installation can be *slightly* easier.
This sort of behaviour towards non-WMF extension developers is interesting and if your objective is to alienate (as with the attitude above) volunteer developers then your on the right path.
You are free to use composer.json to manage extensions, in which case you should version it in SCM. There's no conflict here. We did not favor one use-case over another; we went with the path that coheres with the design of Composer, as explicitly discussed in fantastic detail in its documentation, bringing MediaWiki in line with every other significant application or framework that uses Composer that I could find.
We're not so far down a path that we can't change course, but I've yet to see you rebut any of the points I raised in my commit message accompanying change I3e7c668ee[0] or articulate a coherent alternative.
As for the accusation that the current approach favors the WMF, it's almost not worth responding to. We don't even intend to use Composer in production; all the momentum behind the recent work around Composer integration has in mind how MediaWiki fits with the broader open-source ecosystem.
Hey,
As for the accusation that the current approach favors the WMF, it's almost
not worth responding to.
It seems like what James is getting at is the core community, not the WMF. The problem being that several people seem to thing that the concerns raised are "not relevant" and "not worth responding to".
Composer is simply not meant to be used in the way it was shoe-horned into
MediaWiki.
Starwman. I happen to have discussed the situation and the approach with the main people behind Composer in person, as well as gone over details with contributors on IRC. They did not seem to share your opinion.
I’m not going to re-explain this every time because it is in multiple
places on Bugzilla and in this mailing list.
Who is asking you to re-explain this? What I want is you and others to stop dismissing the concerns raised and coming up with a solution for the problems caused, rather then repeating the same lines over and over again.
We are not going to misuse libraries and hack together MediaWiki just so
extension installation can be *slightly* easier.
In other words, you are fine with breaking existing support, and screwing over both users and developers of extensions such as SMW. In case of SMW, the different is not slight, as it uses libraries itself.
Cheers
-- Jeroen De Dauw - http://www.bn2vs.com Software craftsmanship advocate Evil software architect at Wikimedia Germany ~=[,,_,,]:3
Starwman. I happen to have discussed the situation and the approach with the main people behind Composer in person, as well as gone over details with contributors on IRC. They did not seem to share your opinion. Since we’re throwing out logical fallacies: argumentum ab auctoritate.
Like I’ve *already explained*…
Here is the problem: we need composer.json and composer.lock to be version controlled so that MediaWiki core can manage its own dependencies rather than using git submodules or hard-copying source trees into the repository, which everybody agrees are not sustainable solutions. However, third parties want to be able to install extensions via Composer, which, while not the purpose of Composer, is technically feasible. These two ideas conflict.
Here is the solution: rather than using Composer as a package management system when, in reality, it is a dependency management system, we use Composer to properly maintain core, and then do one of the following: 1) implement our own extension installation system from scratch, or 2) change and/or extend Composer so that it supports a plugin system. I personally recommend the latter, and there are upstream bug reports open concerning it.
-- Tyler Romeo 0xC86B42DF
On Wed, Jun 4, 2014 at 9:53 AM, Tyler Romeo tylerromeo@gmail.com wrote:
Here is the solution: rather than using Composer as a package management system when, in reality, it is a dependency management system, we use Composer to properly maintain core, and then do one of the following: 1) implement our own extension installation system from scratch, or 2) change and/or extend Composer so that it supports a plugin system. I personally recommend the latter, and there are upstream bug reports open concerning it.
As a temporary workaround, maybe we could create extensions/composer-example.json which could be used for extension registration by running composer in that directory?
As a temporary workaround, maybe we could create extensions/composer-example.json which could be used for extension registration by running composer in that directory? Yes. That would work. All we would need to do is add the following into Setup.php:
include “$IP/extensions/vendor/autoload.php”;
(Obviously we’d also have a file_exists check…)
-- Tyler Romeo 0xC86B42DF
On Thu, May 29, 2014 at 11:27 AM, Bryan Davis bd808@wikimedia.org wrote:
My logging changes [0][1][2][3] are getting closer to being mergeable (the first has already been merged). Tony Thomas' Swift Mailer change [4] is also progressing. Both sets of changes introduce the concept of specifying external library dependencies, both required and suggested, to mediawiki/core.git via composer.json. Composer can be used by people directly consuming the git repository to install and manage these dependencies. I gave a example set of usage instructions in the commit message for my patch that introduced the dependency on PSR-3 [0]. In the production cluster, on Jenkins job runners and in the tarball releases we will want a different solution.
My idea of how to deal with this is to create a new gerrit repository (mediawiki/core/vendor.git?) that contains a composer.json file similar to the one I had in patch set 7 of my first logging patch [5]. This composer.json file would be used to tell Composer the exact versions of libraries to download. Someone would manually run Composer in a checkout of this repository and then commit the downloaded content, composer.lock file and generated autoloader.php to the repository for review. We would then be able to branch and use this repository as git submodule in the wmf/1.2XwmfY branches that are deployed to production and ensure that it is checked out along with mw-core on the Jenkins nodes. By placing this submodule at $IP/vendor in mw-core we would be mimicking the configuration that direct users of Composer will experience. WebStart.php already includes $IP/vendor/autoload.php when present so integration with the rest of wm-core should follow from that.
The proposed repository has been created [0] and has an initial set of proposed additions pending review [1].
There is still some ongoing internal discussion about the best way to verify that included libraries are needed and that security patches are watched for and applied from upstream. Chris Steipp is awesome, but it would be quite an additional burden to hang these thousands of new lines of code around his neck as yet another burden to bear. One current theory is that need should be determined by the RFC process and security support would need to be provided by a "sponsor" of the library.
[0]: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/core/vendor [1]: https://gerrit.wikimedia.org/r/#/projects/mediawiki/core/vendor,dashboards/d...
Bryan
On Fri, May 30, 2014 at 3:56 PM, Bryan Davis bd808@wikimedia.org wrote:
There is still some ongoing internal discussion about the best way to verify that included libraries are needed and that security patches are watched for and applied from upstream. Chris Steipp is awesome, but it would be quite an additional burden to hang these thousands of new lines of code around his neck as yet another burden to bear. One current theory is that need should be determined by the RFC process and security support would need to be provided by a "sponsor" of the library.
As long as those libraries are installed via Composer, and well-maintained, something like VersionEye https://www.versioneye.com/ could take on a big part of that burden.
wikitech-l@lists.wikimedia.org