In CR comments on https://gerrit.wikimedia.org/r/#/c/135290/ it has been proposed that we make a git clone of the MW core not be installable until $IP/vendor is populated somehow -- either by separately cloning the mediawiki/core/vendor project, or preferably by running composer to obtain dependencies.
I have suggested, as a compromise, to make the vendor directory be a submodule pointing to mediawiki/core/vendor. Then users can either run "git submodule update --init" to obtain dependencies, or they can omit submodule initialisation and instead run composer.
I would like to hear more comments on this.
-- Tim Starling
On 11/06/14 02:30, Tim Starling wrote:
In CR comments on https://gerrit.wikimedia.org/r/#/c/135290/ it has been proposed that we make a git clone of the MW core not be installable until $IP/vendor is populated somehow -- either by separately cloning the mediawiki/core/vendor project, or preferably by running composer to obtain dependencies.
I have suggested, as a compromise, to make the vendor directory be a submodule pointing to mediawiki/core/vendor. Then users can either run "git submodule update --init" to obtain dependencies, or they can omit submodule initialisation and instead run composer.
I would like to hear more comments on this.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
That's a lot of discussion, and I'm not sure which bits are actually relevant here, so to clarify - are you saying that it's been proposed that MediaWiki not be installable from git without extra steps? If so, what would be the purpose of that?
-I
there is currently a patch in gerrit which would do exactly that if merged.
On Tue, Jun 10, 2014 at 11:22 PM, Isarra Yos zhorishna@gmail.com wrote:
On 11/06/14 02:30, Tim Starling wrote:
In CR comments on https://gerrit.wikimedia.org/r/#/c/135290/ it has been proposed that we make a git clone of the MW core not be installable until $IP/vendor is populated somehow -- either by separately cloning the mediawiki/core/vendor project, or preferably by running composer to obtain dependencies.
I have suggested, as a compromise, to make the vendor directory be a submodule pointing to mediawiki/core/vendor. Then users can either run "git submodule update --init" to obtain dependencies, or they can omit submodule initialisation and instead run composer.
I would like to hear more comments on this.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
That's a lot of discussion, and I'm not sure which bits are actually relevant here, so to clarify - are you saying that it's been proposed that MediaWiki not be installable from git without extra steps? If so, what would be the purpose of that?
-I
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 11/06/14 03:24, John wrote:
there is currently a patch in gerrit which would do exactly that if merged.
Why?
This sounds like a really bad idea, but maybe it isn't, and as a sysadmin and developer I need to know what it even is so I can decide if I should flip the fuck out. Or not.
-I
My suggestion, "Flip the fuck out" Its a really bad idea that wasnt thought though, If users want SwiftMailer support it should be done in an extension, and not in core
On Tue, Jun 10, 2014 at 11:30 PM, Isarra Yos zhorishna@gmail.com wrote:
On 11/06/14 03:24, John wrote:
there is currently a patch in gerrit which would do exactly that if merged.
Why?
This sounds like a really bad idea, but maybe it isn't, and as a sysadmin and developer I need to know what it even is so I can decide if I should flip the fuck out. Or not.
-I
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 11/06/14 03:31, John wrote:
My suggestion, "Flip the fuck out" Its a really bad idea that wasnt thought though, If users want SwiftMailer support it should be done in an extension, and not in core
On Tue, Jun 10, 2014 at 11:30 PM, Isarra Yos zhorishna@gmail.com wrote:
Okay, so I'm completely lost. Is the idea here that SwiftMailer is some shiny new thing that introduces new features, but also new dependencies and whatnot, and that MediaWiki as a whole won't work without the dependencies involved?
If so, does it really need to be in core? And if it doesn't, why isn't it an extension?
And even if it does need to be in core, why should it mean MediaWiki should no longer be installable from git? MediaWiki already has dependencies (which are generally met for all wikis when several are already installed), and the installation process either allows you to progress or not depending on whether or not they are met, but it does at least start out of the box. Why should this be any different? Is this really a sane transition/requirement for developers running test instances, for third-party projects running from git, and other users?
And, again, what's the purpose of it in the first place?
-L
There is zero reason that this shouldnt be in an extension.
Basically a few users want to install a shinny new toy called swiftmailer into core, just because its shiny. In doing so they add a complexity and headache. Such a addition should be done as an extension
On Tue, Jun 10, 2014 at 11:55 PM, Isarra Yos zhorishna@gmail.com wrote:
On 11/06/14 03:31, John wrote:
My suggestion, "Flip the fuck out" Its a really bad idea that wasnt thought though, If users want SwiftMailer support it should be done in an extension, and not in core
On Tue, Jun 10, 2014 at 11:30 PM, Isarra Yos zhorishna@gmail.com wrote:
Okay, so I'm completely lost. Is the idea here that SwiftMailer is some shiny new thing that introduces new features, but also new dependencies and whatnot, and that MediaWiki as a whole won't work without the dependencies involved?
If so, does it really need to be in core? And if it doesn't, why isn't it an extension?
And even if it does need to be in core, why should it mean MediaWiki should no longer be installable from git? MediaWiki already has dependencies (which are generally met for all wikis when several are already installed), and the installation process either allows you to progress or not depending on whether or not they are met, but it does at least start out of the box. Why should this be any different? Is this really a sane transition/requirement for developers running test instances, for third-party projects running from git, and other users?
And, again, what's the purpose of it in the first place?
-L
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 2014-06-10, 8:59 PM, John wrote:
There is zero reason that this shouldnt be in an extension.
Basically a few users want to install a shinny new toy called swiftmailer into core, just because its shiny. In doing so they add a complexity and headache. Such a addition should be done as an extension
Sorry but these strawmen are quite annoying, and frankly disrespectful to the developers trying to improve core.
SwiftMailer is not some toy or shiny trinket, it is a serious and well maintained library dedicated to sending email from PHP.
Adding it to core is not some new feature that would be better in an extension. It is a serious improvement to our core handling of email within PHP that replaces our crude UserMailer code.
Our current UserMailer code is ridiculous when you look at it.
By default our UserMailer code will just use php's creaky mail() function. Which has some 'features' like: - Unpredictable message headers - Lack of feedback regarding delivery failures - And this beautiful comment in our codebase: # PHP's mail() implementation under Windows is somewhat shite, and # can't handle "Joe Bloggs joe@bloggs.com" format email addresses, # so don't bother generating them
If you want to send email a different way, you could of course instead use $wgSMTP which: - First requires you to install PEAR Mail. - Who the hell uses PEAR anymore! - And don't forget, PEAR is a tool that installs modules globally and is difficult to impossible to use without shell and even admin access. - ;) And this is the hell we put tarball users through, not devs. - This PEAR Mail library we're relying on to handle all users who can't use mail() or don't want it's features, here's the dev page: - http://pear.php.net/package/Mail - Current stable 1.2.0, released in March of 2010 - Next release, 1.2.1 scheduled release date in January of 2011, never released. - ((In other words, the library we're entrusting all our SMTP handling to is practically dead and no longer maintained, Whoops.)) - Admittedly if you hunt down PEAR Mail's github repo there has been a bit of activity: - https://github.com/pear/Mail/commits/trunk - But none of this will be installed by pear, it'll only install old code from 2010 missing fixes for a number of bugs in PEAR Mail - The majority of changes in 2014 have been minor/trivial things (adding travis builds, whitespace fixes) - And, ;) making PEAR Mail installable via Composer *snicker*
And to sprinkle this all off, because mail() and PEAR Mail are so different, half the code in UserMailer::send() is split into two different large code paths, which is a recipe for maintenance headaches.
Using Swift Mailer on the other hand: - The library has ample development and maintenance going on: https://github.com/swiftmailer/swiftmailer/commits/master - SMTP and mail() are abstracted away into transports, so instead of two large (unmaintained?) code paths we just craft and send an email in one code path, a well maintained library takes care of the difference between transports. - In the future we can move away from using mail() and add the ability to use sendmail as a transport directly, without the bugs (in theory I think it would even be possible to try swapping a different transport in place of mail() automatically). - All this gets bundled into the tarball directly, so $wgSMTP now works out of the box and doesn't require installation of something that in some situations is impossible to install.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
On 11/06/14 16:18, Daniel Friesen wrote:
- ((In other words, the library we're entrusting all our SMTP handling
to is practically dead and no longer maintained, Whoops.))
Only 3 open bugs though. Sometimes code just keeps working for decades, even without being maintained. Some might even call this a goal.
And to sprinkle this all off, because mail() and PEAR Mail are so different, half the code in UserMailer::send() is split into two different large code paths, which is a recipe for maintenance headaches.
It's only 50 lines of code each way, which is not quite enough to give me a headache. It could use refactoring, but it wouldn't take long to refactor 100 lines of code.
- All this gets bundled into the tarball directly, so $wgSMTP now works
out of the box and doesn't require installation of something that in some situations is impossible to install.
That is a nice feature, yes.
-- Tim Starling
On 2014-06-11, 12:07 AM, Tim Starling wrote:
On 11/06/14 16:18, Daniel Friesen wrote:
- ((In other words, the library we're entrusting all our SMTP handling
to is practically dead and no longer maintained, Whoops.))
Only 3 open bugs though. Sometimes code just keeps working for decades, even without being maintained. Some might even call this a goal.
Well yes, there are only a few (actually 4 not 3) open bugs: http://pear.php.net/bugs/search.php?cmd=display&package_name%5B%5D=Mail&...
Although there's also this: http://pear.php.net/bugs/roadmap.php?package=Mail&roadmapdetail=1.2.1#a1...
Because 1.2.1 was never released to pear there are 6 extra already fixed bugs which everyone using PEAR Mail are still affected by.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
On 11/06/14 16:18, Daniel Friesen wrote:
- Current stable 1.2.0, released in March of 2010
Note that the SMTP part is implemented in Net_SMTP, which was last released in 2013, and the MIME printer part (which we don't currently use) is in Mail_Mime, last released May 2014.
Swift, by contrast, is a 43 KLOC monolith which does all three things and a kitchen sink of other bits and pieces, like its own dependency injection system (Swift_DependencyContainer) supported by a custom autoloader.
- And, ;) making PEAR Mail installable via Composer *snicker*
You snicker at the idea that the maintainers of PEAR Mail are not soldiers in a pitched battle for the minds of developers, PEAR on one side and Composer on the other? Installer neutrality doesn't seem so strange to me.
Note that the PEAR core itself is installable via Composer, and has a composer.json in its git source tree. Net_SMTP and Mail_Mime also have a composer.json.
Maybe we should think about installing PEAR Mail in the vendor directory using Composer, instead of Swift.
-- Tim Starling
On 2014-06-11, 10:55 PM, Tim Starling wrote:
On 11/06/14 16:18, Daniel Friesen wrote:
- And, ;) making PEAR Mail installable via Composer *snicker*You snicker at the idea that the maintainers of PEAR Mail are not soldiers in a pitched battle for the minds of developers, PEAR on one side and Composer on the other? Installer neutrality doesn't seem so strange to me.
Oh no, that's not what the snicker is about.
I was replying to an email completely against the introduction of composer and Swift Mailer to replace the current core UserMailer status quo. ;) So I was snickering at the irony that one of the few changes made to the library we currently use, is support for the very installer that the email I was replying to was opposing.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
It seems worth looking into PEAR mail in my opinion. There something to be said for a certain minimalism in libraries.
-- View this message in context: http://wikimedia.7.x6.nabble.com/Making-a-plain-MW-core-git-clone-not-be-ins... Sent from the Wikipedia Developers mailing list archive at Nabble.com.
On 11/06/14 13:22, Isarra Yos wrote:
That's a lot of discussion, and I'm not sure which bits are actually relevant here, so to clarify - are you saying that it's been proposed that MediaWiki not be installable from git without extra steps? If so, what would be the purpose of that?
In the current change, the purpose is to provide a clean, feature-rich interface which developers can use to send emails.
In particular, there is a mail library called SwiftMailer which provides bounce detection, among other things. Bounce detection would be a nice thing to have. It is proposed that developers use SwiftMailer directly instead of using a MediaWiki wrapper which optionally uses SwiftMailer. This would make the MW core smaller and thus easier to maintain.
-- Tim Starling
This also has knock on impacts elsewhere. BD808 has a patch that uses PSR-log and Monolog for logging. We're starting to move to a model where we recognize that we shouldn't write everything and that things in core have significantly better replacements in the wider PHP community. It doesn't make sense to keep maintaining the vastly inferior core components when more and more core and extensions are going to want to rely on the newer interfaces and features.
The question Tim posed in the commit comes down to: * Do we bundle third party dependencies, or * Do we allow composer to do what it was designed to do and manage the dependencies for us
~Matt Walker Wikimedia Foundation Fundraising Technology Team
On Tue, Jun 10, 2014 at 9:49 PM, Tim Starling tstarling@wikimedia.org wrote:
On 11/06/14 13:22, Isarra Yos wrote:
That's a lot of discussion, and I'm not sure which bits are actually relevant here, so to clarify - are you saying that it's been proposed that MediaWiki not be installable from git without extra steps? If so, what would be the purpose of that?
In the current change, the purpose is to provide a clean, feature-rich interface which developers can use to send emails.
In particular, there is a mail library called SwiftMailer which provides bounce detection, among other things. Bounce detection would be a nice thing to have. It is proposed that developers use SwiftMailer directly instead of using a MediaWiki wrapper which optionally uses SwiftMailer. This would make the MW core smaller and thus easier to maintain.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 11.06.2014 9:01, Matthew Walker wrote:
This also has knock on impacts elsewhere. BD808 has a patch that uses PSR-log and Monolog for logging. We're starting to move to a model where we recognize that we shouldn't write everything and that things in core have significantly better replacements in the wider PHP community. It doesn't make sense to keep maintaining the vastly inferior core components when more and more core and extensions are going to want to rely on the newer interfaces and features.
PSR loader is ok. PSR "standard" of using four spaces instead of tabs for indentation is strange. That prevents from easily adjusting the indentation (good editors can visually render tabs like N-spaces according to user preference) and has another drawbacks.
The question Tim posed in the commit comes down to:
- Do we bundle third party dependencies, or
- Do we allow composer to do what it was designed to do and manage the
dependencies for us
composer can use git to fetch dependencies, at least it does so when I developed with symfony2. https://getcomposer.org/doc/05-repositories.md Dmitriy
On 11/06/14 05:01, Matthew Walker wrote:
This also has knock on impacts elsewhere. BD808 has a patch that uses PSR-log and Monolog for logging. We're starting to move to a model where we recognize that we shouldn't write everything and that things in core have significantly better replacements in the wider PHP community. It doesn't make sense to keep maintaining the vastly inferior core components when more and more core and extensions are going to want to rely on the newer interfaces and features.
The question Tim posed in the commit comes down to:
- Do we bundle third party dependencies, or
- Do we allow composer to do what it was designed to do and manage the
dependencies for us
~Matt Walker Wikimedia Foundation Fundraising Technology Team
Problem with composer is it doesn't actually work for all use cases (farms in particular can be problematic, even if the wmf did get it working), so depending on it isn't really a good idea unless there is a sane fallback. What folks have mentioned as alternatives so far (submodules in core etc) do not sound sane.
I'm all for having better handling for things in core, but we need a sane way to include them, and sane defaults that don't require weird dependencies and bending over backwards when all you want to do is clone the thing and get set up.
-I
On Wed, Jun 11, 2014 at 7:20 PM, Isarra Yos zhorishna@gmail.com wrote:
Problem with composer is it doesn't actually work for all use cases (farms in particular can be problematic, even if the wmf did get it working), so depending on it isn't really a good idea unless there is a sane fallback. What folks have mentioned as alternatives so far (submodules in core etc) do not sound sane.
Which use cases are you referring to exactly? (I'm curious as to how composer does not work with farms.)
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science
On 11/06/14 23:21, Tyler Romeo wrote:
On Wed, Jun 11, 2014 at 7:20 PM, Isarra Yos zhorishna@gmail.com wrote:
Problem with composer is it doesn't actually work for all use cases (farms in particular can be problematic, even if the wmf did get it working), so depending on it isn't really a good idea unless there is a sane fallback. What folks have mentioned as alternatives so far (submodules in core etc) do not sound sane.
Which use cases are you referring to exactly? (I'm curious as to how composer does not work with farms.)
I'm afraid the sysadmins responsible for the decisions at ShoutWiki and Uncyclomedia would have to speak for themselves as to details; I only know it didn't work for either, and that such is something to consider. (Apparently there was an rfc on this - did nobody bring it up there? Or did they even know about it? Or am I thinking of something else?)
-I
On 11/06/14 14:49, I wrote:
In particular, there is a mail library called SwiftMailer which provides bounce detection, among other things. Bounce detection would be a nice thing to have.
Sorry, it seems I was misled on this. SwiftMailer does not have any special handling for bounces, judging by the code review I have done just now and
http://pookey.co.uk/wordpress/archives/183-using-verp-with-swiftmailer-symfony-and-exim
All it has is a setReturnPath() method, and of course we have equivalent code in UserMailer.php already:
$headers['Return-Path'] = $from->address;
With VERP this would become:
$headers['Return-Path'] = $this->makeReturnPath( $from );
or some such. With SwiftMailer, you could instead have:
$message->setReturnPath( $this->makeReturnPath( $from ) );
...with identical implementation of makeReturnPath(). SwiftMailer does not give you any help in receiving bounces. It has no MIME parsers or clients for POP, IMAP, MTA pipes, etc.
On [[Talk:VERP]], Tyler suggested writing a SwiftMailer plugin to "implement VERP", by which he apparently means a plugin which would serialize the relevant parameters and calculate an HMAC, and then set the return path.
Presumably this would look like:
$verp = new Swift_Plugins_VerpPlugin( array( 'wiki' => wfWikiId(), 'ua' => 'UserMailer', 'time' => wfTimestampNow() ), $secretKey ) ); $mailer->registerPlugin( $verp ); $mailer->send( $message );
With Swift_Plugins_VerpPlugin being along the lines of:
class Swift_Plugins_VerpPlugin { function __construct( $context ) { $this->context = $context; }
function beforeSendPerformed( $evt ) { $message = $evt->getMessage(); $message->setReturnPath( $this->makeReturnPath( $message->getReturnPath() ) ); } ... }
...with makeReturnPath() being the serialization code, equivalent to my suggestion of UserMailer::makeReturnPath() above.
On [[Talk:VERP]], Jeff Green was fairly skeptical about the value of doing it this way, and I guess I am too. You have to wonder where the boundary is between abstraction and needless complexity. And where do you put the receive code? Surely the return path parser and the return path printer should be maintained together. SwiftMailer is basically just an SMTP client. I would think that a bounce processing feature as a whole would be an elegant thing to put a module boundary around, rather than having half of it implemented as an SMTP client plugin and the other half implemented as a MediaWiki-specific CLI interface for an MTA pipe.
In other news, I found a serious security vulnerability in SwiftMailer and have reported it upstream.
-- Tim Starling
On 2014-06-10, 7:30 PM, Tim Starling wrote:
I have suggested, as a compromise, to make the vendor directory be a submodule pointing to mediawiki/core/vendor. Then users can either run "git submodule update --init" to obtain dependencies, or they can omit submodule initialisation and instead run composer.
I can only think of one potential pain point with this. If someone does `composer install` instead of checking out the submodule, won't `git status` complain about that (by displaying it as a modification to core)?
That said, I don't see the point of having mediawiki/core/vendor without making it a submodule, so I'm in favour of adding it.
To clarify, without a submodule link you don't know what the correct commit within mediawiki/core/vendor for the commit of MW you have is tied to: - On REL1_## branches simply cloning mediawiki/core/vendor would bring newer versions of libraries that don't match the one the release was made for. They could be potentially incompatible breaking the wiki, or may have bugfixes that hide bugs encountered by tarball users when you try to verify them. - And it could be worse for individual commits in master. It should be possible to `git bisect` anything within core, so we definitely should make sure every commit knows what commit of mediawiki/core/vendor it should be using, rather than only ensuring that each of our git heads has a matching mediawiki/core/vendor head to use.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
What about doing the reasonable thing and leaving core the hell alone? this should be an extension and not shoved into core.
On Tue, Jun 10, 2014 at 11:43 PM, Daniel Friesen <daniel@nadir-seen-fire.com
wrote:
On 2014-06-10, 7:30 PM, Tim Starling wrote:
I have suggested, as a compromise, to make the vendor directory be a submodule pointing to mediawiki/core/vendor. Then users can either run "git submodule update --init" to obtain dependencies, or they can omit submodule initialisation and instead run composer.
I can only think of one potential pain point with this. If someone does `composer install` instead of checking out the submodule, won't `git status` complain about that (by displaying it as a modification to core)?
That said, I don't see the point of having mediawiki/core/vendor without making it a submodule, so I'm in favour of adding it.
To clarify, without a submodule link you don't know what the correct commit within mediawiki/core/vendor for the commit of MW you have is tied to:
- On REL1_## branches simply cloning mediawiki/core/vendor would bring
newer versions of libraries that don't match the one the release was made for. They could be potentially incompatible breaking the wiki, or may have bugfixes that hide bugs encountered by tarball users when you try to verify them.
- And it could be worse for individual commits in master. It should be
possible to `git bisect` anything within core, so we definitely should make sure every commit knows what commit of mediawiki/core/vendor it should be using, rather than only ensuring that each of our git heads has a matching mediawiki/core/vendor head to use.
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hi,
from my point of view this discussion mixes two or three different questions:
* Do we want core to regularly use external libraries (which are installed via one git submodule or composer)? * Should Swiftmailer be used in core? * (How) Do we want to handle this for plain git cloning to be usable?
I think the first question is sort of answered, although maybe not everybody committed on that.
The second question is something different entirely. From my point of view it doesn't make any sense to discuss this here, because (considering how we IMHO answered the first question) we will have a required dependency eventually, even if that is not going to be Swiftmailer.
The third question is what Tim actually asked, and it's an interesting, technical question we should focus on. I like Tim's suggestion of having an optional git submodule. The installer could then show an error if vendor/ is missing proposing to either run "composer install" or "git submodule update --init". Do we also have to consider composer.lock?
Regards, Adrian
Just to add to the technical stuff:- * We made a local repo to fork the original swiftmailer repo at https://github.com/wikimedia/mediawiki-core-vendor-swiftmailer/tree/5.2.0-pa..., and our https://gerrit.wikimedia.org/r/#/c/137538/ makes composer take up the files from our repo and not the upstream one, which is for the development end. * Our VERP project is going on at https://gerrit.wikimedia.org/r/#/c/138655/
Thanks, Tony Thomas http://tttwrites.in FOSS@Amrita http://foss.amrita.ac.in
*"where there is a wifi, there is a way"*
On Wed, Jun 11, 2014 at 1:10 PM, Adrian Lang adrian.lang@wikimedia.de wrote:
Hi,
from my point of view this discussion mixes two or three different questions:
- Do we want core to regularly use external libraries (which are
installed via one git submodule or composer)?
- Should Swiftmailer be used in core?
- (How) Do we want to handle this for plain git cloning to be usable?
I think the first question is sort of answered, although maybe not everybody committed on that.
The second question is something different entirely. From my point of view it doesn't make any sense to discuss this here, because (considering how we IMHO answered the first question) we will have a required dependency eventually, even if that is not going to be Swiftmailer.
The third question is what Tim actually asked, and it's an interesting, technical question we should focus on. I like Tim's suggestion of having an optional git submodule. The installer could then show an error if vendor/ is missing proposing to either run "composer install" or "git submodule update --init". Do we also have to consider composer.lock?
Regards, Adrian
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Le 11/06/2014 04:30, Tim Starling a écrit :
In CR comments on https://gerrit.wikimedia.org/r/#/c/135290/ it has been proposed that we make a git clone of the MW core not be installable until $IP/vendor is populated somehow -- either by separately cloning the mediawiki/core/vendor project, or preferably by running composer to obtain dependencies.
I have suggested, as a compromise, to make the vendor directory be a submodule pointing to mediawiki/core/vendor. Then users can either run "git submodule update --init" to obtain dependencies, or they can omit submodule initialisation and instead run composer.
I would like to hear more comments on this.
Hello,
I would prefer us to avoid embedding third party libraries directly in core: - the repository is already big enough as it is (recently Chad proposed to drop history to shrink the repo). - we might be tempted to have local hack and forget to push them back to upstream
Assuming the above, we would require an extra step and I am fine with it. For people running mediawiki out of a git clone that should not be too much of an hassle, the rest of the user base uses the tarball or a package which could both embed all the third party libraries.
I do not think we should mix both system and should make a choice between git submodule and composer.
A rough comparison on top of my mind, should probably start up a formal RFC about it though they are probably some going on.
== git submodule ==
+ command already available + already used by Wikimedia to handle extensions dependencies in the wmf branches + let us review the code + ability to patch third parties - require to fully clone each repositories - version tracked by git sha1 in .gitmodules instead of a version number
== composer ==
+ generates autoloader entries + has a post install system which could be used to inject settings in LocalSettings.php + could be used to handle extensions dependencies - depends on upstream to incorporate our patches - needs to install composer - might not be used as-is on Wikimedia cluster
Before we make any decision, I would love to hear from Bryan Davis (he is on vacations this week) and Jeroen De Dauw who already put a lot of efforts on dependencies management.
cheers,
On Wed, Jun 11, 2014 at 1:58 PM, Antoine Musso hashar+wmf@free.fr wrote:
A rough comparison on top of my mind, should probably start up a formal RFC about it though they are probably some going on.
Please find Bryan's RFC here:- https://www.mediawiki.org/wiki/Requests_for_comment/Composer_managed_librari...
Thanks, Tony Thomas http://tttwrites.in FOSS@Amrita http://foss.amrita.ac.in
*"where there is a wifi, there is a way"*
On Wed, Jun 11, 2014 at 4:28 AM, Antoine Musso hashar+wmf@free.fr wrote:
- depends on upstream to incorporate our patches
- might not be used as-is on Wikimedia cluster
I would just like to point out that you can override where Composer finds packages using the composer.json file. In other words, if we wanted to force Composer to use WMF's git repositories to check out dependencies, that is entirely possible. In the end, Composer uses git to check out the packages anyway.
https://getcomposer.org/doc/04-schema.md#repositories
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science
OK, now that I have some more time, I will expand on what I’ve already said in the Gerrit patch.
== Introduction ==
The WMF does not have that many developers. And it’s not that they haven’t hired enough or anything like that. It’s just they do not have that many people in general. As of last year, they had under 200 employees, total. I work at an enterprise software company, and we have 200 developers alone on the app/dev team, let alone the additional people (including myself) in the architecture group, in the business analyst team, in management, etc. To think that under 200 people can develop a 400,000+ line codebase is insane.
That’s why the WMF gets help from the open source community. Not only because people love to work on open source projects that benefit the world and humanity, but because such projects *need* volunteers because they simply do not have the billion-dollar operating revenue necessary to develop an enterprise product.
With that said, it is absolutely necessary that the WMF get as much help as it can. Right now it has a lot of help: numerous volunteer developers working on core, working on translatewiki, etc. However, it’s not easy convincing people to join an open source software project. I’m sure Quim and Sumana can attest to this. What we can do, however, is “export” the work.
== Third-party libraries ==
=== How they can help? ===
Third-party libraries, i.e., other open-source software projects, are helpful in that:
1) They have existing communities of interested volunteer (and sometimes even paid) developers. 2) They are structured in an abstract manner, meaning they can be used inside MediaWiki without having extensive knowledge of the library’s internals. 3) They help to remove possibly hundreds if not thousands of lines of code from MediaWiki core that does not need to be there.
The premise is: why reinvent the wheel? Or, to be more specific, why reinvent the wheel by carving a square stone rock? There are libraries out there (such as SwiftMailer, which I’ll get to) that do certain tasks that MediaWiki needs to do. However, they are ten times as comprehensive as MediaWiki’s make-shift implementations, and are maintained a lot more.
=== The qualifications ===
Obviously, we cannot just start throwing in third-party libraries willy-nilly. There are major implications, specifically:
* Possible security issues introduced by the library * Having to submit patches upstream when bugs are discovered
These are real issues. However, if the third-party library we are thinking of including is: 1) popular, i.e., used by other projects, especially commercial and/or commercially-backed projects; 2) uses a sane and reliable code review process and release process; and 3) is stable and unlikely to change drastically, then the library is reliable, and we do not really have to worry about issues.
One argument that people seem to be making is that since the code did not go through *our* review process, the code might be vulnerable or insecure. That is a fallacy. To say that code going through our own review process is more secure than others is ridiculous. Doing a security review of code requires security knowledge, and those with security knowledge (i.e., Chris) are not reviewing every line of code that goes into core.
Long story short, we do need to evaluate what libraries we put in, but if they are reliable it will not be a problem.
== SwiftMailer ==
Now we get to SwiftMailer. Some are calling it a new shiny toy, and like Daniel said, this is a bit insulting to anybody who has even looked at the UserMailer.php code.
With SwiftMailer, the only mail code that will be in MediaWiki will be what is necessary to get the e-mail address of a user. Everything else is handled inside the library. We no longer have to maintain mail-sending code at all. Putting PEAR aside, considering the major annoyances in using PHP’s mail() function, including differences in handling between operating systems, it is a big success if we do not have to deal with the mail() function inside MediaWiki.
-- Tyler Romeo 0xC86B42DF
From: Tyler Romeo tylerromeo@gmail.com Reply: Tyler Romeo tylerromeo@gmail.com Date: June 11, 2014 at 7:21:46 To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Making a plain MW core git clone not be installable
On Wed, Jun 11, 2014 at 4:28 AM, Antoine Musso hashar+wmf@free.fr wrote: - depends on upstream to incorporate our patches - might not be used as-is on Wikimedia cluster
I would just like to point out that you can override where Composer finds packages using the composer.json file. In other words, if we wanted to force Composer to use WMF's git repositories to check out dependencies, that is entirely possible. In the end, Composer uses git to check out the packages anyway.
https://getcomposer.org/doc/04-schema.md#repositories
-- Tyler Romeo Stevens Institute of Technology, Class of 2016 Major in Computer Science
I will mention that any solution short of sucking in the third party dependencies into the main repo (not as a submodule) -- which no one wants to do anyway -- will be slightly awkward to git bisect. Not impossible; the pain is about the same for both main options:
a) in theory git-bisect should adjust submodules to the correct hash. in practice you need to run `git submodule update` after every step in order to check out the appropriate submodule commits.
b) similarly, for composer you need to run a command to update the 3rd party packages, if any dependencies have changed. (for node.js projects, which have similar issues, you need to run `npm install` after every step.
For regressions that are easily found by running a test suite, you can arrange for `git bisect run` to do the appropriate git, composer, or npm command before running the test.
So, it's somewhat awkward, but manageable. And usually the 3rd-party dependencies don't typically change as often as the core code, so this doesn't come up all that often.
Two main benefits of `git submodule`: (1) perhaps one day `git bisect` and `git submodule` will play nicer together; (2) since references are to a git hash, crawling through history is repeatable. One disadvantages: (1) because git allows `.gitmodules` to differ from your local set of module repo sources (in `.git/config`), it is rather too easy to forget to push a submodule commit referenced from the main repo -- although hopefully jenkins will catch errors of this sort,
The main disadvantage of using `composer`/`npm`/etc directly is that you are at the mercy of the upstream package repo to behave well. That is, if the upstream repo allows uploaders to change the tarball associated with version X.Y.Z, you may find it hard to reproduce past configurations. Similarly, if you specify loose package version constraints, you are at the mercy of the upstream maintainers to actually maintain compatibility between minor versions or what-have-you.
Parsoid and some other WMF projects actually use a hybrid approach where we use `npm` and not submodules, but we maintain a separate "deploy" repo which combines the main code base (as a submodule) and specific versions of the third party code. --scott
I'm not a developer so it's perfectly normal that I can't understand anything about your talk; nevertheless, please, remember of KISS principle when building any installing tools for poor, final users. I'm waiting for something like "pip install core".
Alex
2014-06-11 15:58 GMT+02:00 C. Scott Ananian cananian@wikimedia.org:
I will mention that any solution short of sucking in the third party dependencies into the main repo (not as a submodule) -- which no one wants to do anyway -- will be slightly awkward to git bisect. Not impossible; the pain is about the same for both main options:
a) in theory git-bisect should adjust submodules to the correct hash. in practice you need to run `git submodule update` after every step in order to check out the appropriate submodule commits.
b) similarly, for composer you need to run a command to update the 3rd party packages, if any dependencies have changed. (for node.js projects, which have similar issues, you need to run `npm install` after every step.
For regressions that are easily found by running a test suite, you can arrange for `git bisect run` to do the appropriate git, composer, or npm command before running the test.
So, it's somewhat awkward, but manageable. And usually the 3rd-party dependencies don't typically change as often as the core code, so this doesn't come up all that often.
Two main benefits of `git submodule`: (1) perhaps one day `git bisect` and `git submodule` will play nicer together; (2) since references are to a git hash, crawling through history is repeatable. One disadvantages: (1) because git allows `.gitmodules` to differ from your local set of module repo sources (in `.git/config`), it is rather too easy to forget to push a submodule commit referenced from the main repo -- although hopefully jenkins will catch errors of this sort,
The main disadvantage of using `composer`/`npm`/etc directly is that you are at the mercy of the upstream package repo to behave well. That is, if the upstream repo allows uploaders to change the tarball associated with version X.Y.Z, you may find it hard to reproduce past configurations. Similarly, if you specify loose package version constraints, you are at the mercy of the upstream maintainers to actually maintain compatibility between minor versions or what-have-you.
Parsoid and some other WMF projects actually use a hybrid approach where we use `npm` and not submodules, but we maintain a separate "deploy" repo which combines the main code base (as a submodule) and specific versions of the third party code. --scott
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Jun 11, 2014 at 10:42 AM, Alex Brollo alex.brollo@gmail.com wrote:
I'm not a developer so it's perfectly normal that I can't understand anything about your talk; nevertheless, please, remember of KISS principle when building any installing tools for poor, final users. I'm waiting for something like "pip install core".
Just to clarify on the implications of using Composer, specifically:
1) If you are using MediaWiki through tarball releases (or rather, anything other than using git), this will *not* affect you. Tarballs will ship with the dependencies pre-loaded, so nothing changes.
2) If you are using MediaWiki through git, the extra step you will need to do to install MediaWiki is:
curl -sS https://getcomposer.org/installer | php ./composer.phar install
And hopefully, if the user has enough knowledge to clone a git repository, they should understand command line and how to run these two basic commands. (As a side note, you can install composer globally by moving composer.phar to /usr/bin/local/composer, which makes thing simpler.)
Of course, all of this will be thoroughly documented.
Additionally, it is also possible to just integrate Composer into the MediaWiki installer, and have the normal install process load the dependencies on its own. Of course, this assumes the web server has write permissions to the install directory.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science
On Wed, Jun 11, 2014 at 10:51 AM, Tyler Romeo tylerromeo@gmail.com wrote:
curl -sS https://getcomposer.org/installer | php
... That's just awful.
On Wed, Jun 11, 2014 at 10:58 AM, Tyler Romeo tylerromeo@gmail.com wrote:
On Wed, Jun 11, 2014 at 10:56 AM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
... That's just awful.
How so?
Well, it makes *me* wince because you're directing people to pull code over the network and feed it straight to the PHP interpreter, probably as root, without inspecting it first. And the site is happy to send it to you via plain HTTP, which means a one-character typo gives an active attacker a chance to pwn your entire installation.
No, nobody bothers to read all the code they just checked out of Git, but it's integrity-protected by design, independent of the transport channel -- you know that the code you just received is the exact same code everyone else is getting, so you can trust that *someone* did the security audit.
(And yeah, no one does *that* either, which is how we got the OpenSSL fiasco, but computers can't solve that problem.)
zw
On Wed, Jun 11, 2014 at 11:05 AM, Zack Weinberg zackw@cmu.edu wrote:
Well, it makes *me* wince because you're directing people to pull code over the network and feed it straight to the PHP interpreter, probably as root, without inspecting it first. And the site is happy to send it to you via plain HTTP, which means a one-character typo gives an active attacker a chance to pwn your entire installation.
It's over HTTPS. As long as you trust that getcomposer.org is the domain you are looking for, this is really no different than installing via a package manager.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science
Ouch, thanks for wasting a few of my brain cells. This is why do dont add stupid code to core.
My web server doesnt have curl installed, nor does it have /usr/bin/local/
You havent bothered to think your code through. Why dont you un-fuck your code, configure it as an extension and go from there? at that point you can find out exactly how many site your going to break. Once you have a stable reviewed extension we can *think* about merging it to core.
On Wed, Jun 11, 2014 at 11:21 AM, Tyler Romeo tylerromeo@gmail.com wrote:
On Wed, Jun 11, 2014 at 11:05 AM, Zack Weinberg zackw@cmu.edu wrote:
Well, it makes *me* wince because you're directing people to pull code over the network and feed it straight to the PHP interpreter, probably as root, without inspecting it first. And the site is happy to send it to you via plain HTTP, which means a one-character typo gives an active attacker a chance to pwn your entire installation.
It's over HTTPS. As long as you trust that getcomposer.org is the domain you are looking for, this is really no different than installing via a package manager.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Can we kill the subthread dealing with the awful "pipe the output of curl to php" install for composer? It's evilness is not really on topic (not until we start writing suggested install directions in the wiki). As Chad noted, there are sane-sysadmin ways to install composer. I think it would be more productive to continue discussing how we want to handle third-party dependencies, rather than arguing over install instructions. --scott
On Wed, Jun 11, 2014 at 11:21 AM, Tyler Romeo tylerromeo@gmail.com wrote:
On Wed, Jun 11, 2014 at 11:05 AM, Zack Weinberg zackw@cmu.edu wrote:
Well, it makes *me* wince because you're directing people to pull code over the network and feed it straight to the PHP interpreter, probably as root, without inspecting it first. And the site is happy to send it to you via plain HTTP, which means a one-character typo gives an active attacker a chance to pwn your entire installation.
It's over HTTPS. As long as you trust that getcomposer.org is the domain you are looking for, this is really no different than installing via a package manager.
*-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Jun 11, 2014 at 11:21 AM, Tyler Romeo tylerromeo@gmail.com wrote:
It's over HTTPS. As long as you trust that getcomposer.org is the domain you are looking for, this is really no different than installing via a package manager.
Nothing stops you from installing it over insecure HTTP. (I filed https://github.com/composer/composer/issues/3047 for that.)
But this is bad practice even with HTTPS; you're relying on *transport* integrity/authenticity to secure *document* authenticity. Yeah, we do that all the time on today's Web, but software installation is (I don't think this is hyperbole) more security-critical than anything else and should be held to higher standards. In this case, there should be an independently verifiable (i.e. not tied to the TLS PKI) PGP signature on the installer and people should be instructed to check that before executing it.
Note that Git submodules do this for you automatically, because the revision hash is unforgeable.
zw
No one has really addressed the point of making this an extension and not adding the excessive overhead to core.
Especially for something that may have such a wide impact.
Thanks Zack for actually explaining the reasoning to me, rather than trying to insult my intelligence and then use it as an argument against the proposal. -- Tyler Romeo 0xC86B42DF
From: Zack Weinberg zackw@cmu.edu Reply: Wikimedia developers wikitech-l@lists.wikimedia.org Date: June 11, 2014 at 11:47:34 To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Making a plain MW core git clone not be installable
Nothing stops you from installing it over insecure HTTP. (I filed https://github.com/composer/composer/issues/3047%C2%A0for that.)
But this is bad practice even with HTTPS; you're relying on *transport* integrity/authenticity to secure *document* authenticity. Yeah, we do that all the time on today's Web, but software installation is (I don't think this is hyperbole) more security-critical than anything else and should be held to higher standards. In this case, there should be an independently verifiable (i.e. not tied to the TLS PKI) PGP signature on the installer and people should be instructed to check that before executing it.
Note that Git submodules do this for you automatically, because the revision hash is unforgeable.
Fwiw, that's not required. Lots of packages exist already. Even if you don't have a package for your chosen distro you can easily install it to your $PATH in less scary ways.
-Chad On Jun 11, 2014 7:56 AM, "Brad Jorsch (Anomie)" bjorsch@wikimedia.org wrote:
On Wed, Jun 11, 2014 at 10:51 AM, Tyler Romeo tylerromeo@gmail.com wrote:
curl -sS https://getcomposer.org/installer | php
... That's just awful. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Wed, Jun 11, 2014 at 4:28 AM, Antoine Musso hashar+wmf@free.fr wrote:
== git submodule ==
- command already available
- already used by Wikimedia to handle extensions dependencies in the wmf branches
- let us review the code
- ability to patch third parties
- require to fully clone each repositories
Couldn't you do a shallow clone, if the disk space or bandwidth is a concern?
- version tracked by git sha1 in .gitmodules instead of a version number
== composer ==
- generates autoloader entries
- has a post install system which could be used to inject settings in
LocalSettings.php
This could well be a minus if it winds up being done poorly.
+ could be used to handle extensions dependencies
- depends on upstream to incorporate our patches
Well, we could fork if necessary. Although that's seldom fun.
- needs to install composer
This is one of my pet peeves. It's one thing to have to install dependencies to run MediaWiki, but having to install various dependencies just to *install* it? Ugh!
Composer is a first step. Then there's the proposals floating around to make various nodejs services be required (which themselves tend to be a pain because they seem to require dozens of npm packages), which leads to proposals that MediaWiki require puppet/docker/something like that just to be installed because of the overly complex interdependencies. I have nothing against puppet/docker/etc in general, but it seems like overkill for running a simple MediaWiki installation off of a single host.
- might not be used as-is on Wikimedia cluster
Le 11/06/2014 15:51, Brad Jorsch (Anomie) a écrit :
- require to fully clone each repositories
Couldn't you do a shallow clone, if the disk space or bandwidth is a concern?
The git submodule add|update subcommands support --depth to produce shallow clone.
The commit: https://github.com/git/git/commit/275cd184d
Introduced in git v1.8.4 ( git tag --contains 275cd184 )
So yeah that would save up time/bandwith/disk.
Le 11/06/2014 15:51, Brad Jorsch (Anomie) a écrit :
- needs to install composer
This is one of my pet peeves. It's one thing to have to install dependencies to run MediaWiki, but having to install various dependencies just to *install* it? Ugh!
Composer is a first step. Then there's the proposals floating around to make various nodejs services be required (which themselves tend to be a pain because they seem to require dozens of npm packages), which leads to proposals that MediaWiki require puppet/docker/something like that just to be installed because of the overly complex interdependencies. I have nothing against puppet/docker/etc in general, but it seems like overkill for running a simple MediaWiki installation off of a single host.
composer is being more and more used in the PHP world and I see it as essentially replacing Pear that probably nobody is going to regret.
So if one is going to install MediaWiki from git, there is a very high chance he already has composer and the person would definitely be able to figure out how to install it.
For end users, they would rely on tarballs that ships the third parties dependencies.
The orientation toward service oriented architecture (ie bunch of well defined API implemented in whatever language) would surely make it harder and harder to keep supporting third parties. But that is another can of worm.
On 11 Jun 2014, at 22:05, Antoine Musso hashar+wmf@free.fr wrote:
Le 11/06/2014 15:51, Brad Jorsch (Anomie) a écrit :
- needs to install composer
This is one of my pet peeves. It's one thing to have to install dependencies to run MediaWiki, but having to install various dependencies just to *install* it? Ugh!
Composer is a first step. [..]
composer is being more and more used in the PHP world and I see it as essentially replacing Pear that probably nobody is going to regret. [..]
True, composer should be easier to work with than PEAR[1].
However, afaik we never depended on any PEAR packages. We either shipped them (Services_JSON), provided a fallback or made the feature optional / in an extension.
-- Krinkle
[1] Easier to instal because it requires less permissions since it uses local directory. Easier to deploy because of this. And managed with a manifest in composer.json instead of reading a manual of sorts etc.
(anonymous) wrote:
- needs to install composer
This is one of my pet peeves. It's one thing to have to install dependencies to run MediaWiki, but having to install various dependencies just to *install* it? Ugh!
Composer is a first step. [..]
composer is being more and more used in the PHP world and I see it as essentially replacing Pear that probably nobody is going to regret. [..]
True, composer should be easier to work with than PEAR[1].
However, afaik we never depended on any PEAR packages. We either shipped them (Services_JSON), provided a fallback or made the feature optional / in an extension.
-- Krinkle
[1] Easier to instal because it requires less permissions since it uses local directory. Easier to deploy because of this. And managed with a manifest in composer.json instead of reading a manual of sorts etc.
Shouldn't http://pear.php.net/manual/en/installation.shared.php allow installation of PEAR packages in local directories?
Tim
wikitech-l@lists.wikimedia.org