Hi, We were having discussions regarding putting the new SwiftMailer[1] lib in/out of core, after the merge of the change https://gerrit.wikimedia.org/r/#/c/135290. Tyler recommends to add the installer code to the composer.json and not to add the SwiftMailer code to the core. This creates the swiftmailer lib in core/vendors/swiftmailer. After discussing with Bryan (https://dpaste.de/XVkL/raw), it looks like *maintaining a separate repo* for external libraries looked like the best solution, so that it uses composer 'properly', and still works for wmf-deployement -- rather than getting the whole into core. It could be thus deployed via trebuchet to the cluster and the autoloader pulled in in CommonSetings (quoting Bryan). Since the entire mail structure is being reworded to use swiftmailer, and its a crucial dependancy ( There will be no alternate mail systems in UserMailer.php -- as both the SMTP and no SMTP cases are handled effectively by SwiftMailer ), I think we will need to have a separate repo for that. Please go through the patch-set above and comment your opinions on including external libraries.
[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=63483
Thanks, Tony Thomas http://tttwrites.in FOSS@Amrita http://foss.amrita.ac.in
*"where there is a wifi, there is a way"*
Tony Thomas 01tonythomas@gmail.com writes:
We were having discussions regarding putting the newSwiftMailer[1] lib in/out of core, after the merge of the change https://gerrit.wikimedia.org/r/#/c/135290. Tyler recommends to add the installer code to the composer.json and not to add the SwiftMailer code to the core.
Thank you! I think the more use we make of composer, the better.
I do have one concern about the tarballs we produce.
It looks like it will be necessary to pre-populate vendor with composer.json dependencies like this one -- libraries that are essential -- in order to make sure that a tarball installation require downloading anything but the tarball itself.
Thoughts?
On Tue, May 27, 2014 at 2:56 AM, Tony Thomas 01tonythomas@gmail.com wrote:
Hi, We were having discussions regarding putting the new SwiftMailer[1] lib in/out of core, after the merge of the change https://gerrit.wikimedia.org/r/#/c/135290. Tyler recommends to add the installer code to the composer.json and not to add the SwiftMailer code to the core. This creates the swiftmailer lib in core/vendors/swiftmailer. After discussing with Bryan (https://dpaste.de/XVkL/raw), it looks like *maintaining a separate repo* for external libraries looked like the best solution, so that it uses composer 'properly', and still works for wmf-deployement -- rather than getting the whole into core. It could be thus deployed via trebuchet to the cluster and the autoloader pulled in in CommonSetings (quoting Bryan). Since the entire mail structure is being reworded to use swiftmailer, and its a crucial dependancy ( There will be no alternate mail systems in UserMailer.php -- as both the SMTP and no SMTP cases are handled effectively by SwiftMailer ), I think we will need to have a separate repo for that. Please go through the patch-set above and comment your opinions on including external libraries.
Thanks for getting this discussion started Tony. I have "been meaning to" write a very similar email regarding discussions from the Zürich hackathon about my own attempt to pull third-party libraries into mediawiki-core [2]. After talking to Ori, Tyler, Matt Walker and others in Zürich I realized that my desire to put the libraries into mediawiki-core was mostly motivated by convenience for deployment on the production wiki[mp]edia cluster. For a variety of reasons, using composer to download and install libraries directly on the production cluster is undesirable. There is an alternative however that can be clearly seen in the mediawiki/extensions/Parsoid/js/contrib.git repository.
I think we should create a new repository in gerrit ("mediawiki/core/contrib"? Bikeshed as needed) where composer is used to manage importing specific versions of external libraries that are needed for the wiki[mp]edia cluster deployments. This repository can be deployed to the production cluster using Trebuchet or scap and appropriate changes can be made in operations/mediawiki-config.git to ensure that the autoloader can find the external classes.
[2]: https://gerrit.wikimedia.org/r/#/c/119939/
Bryan
That sounds great Bryan.
I think we should create a new repository in gerrit
("mediawiki/core/contrib"? waiting for that repo to show up in gerrit.
On Tue, May 27, 2014 at 7:38 PM, Mark A. Hershberger mah@nichework.com wrote:
Thank you! I think the more use we make of composer, the better.
I do have one concern about the tarballs we produce.
It looks like it will be necessary to pre-populate vendor with composer.json dependencies like this one -- libraries that are essential -- in order to make sure that a tarball installation require downloading anything but the tarball itself.
So, if we can have the swiftmailer repo added, this would be excellent. It might get worse otherwise, as after this shift, MW can 'only' send emails through swift. A hard dependency, as you quoted earlier.
Thanks, Tony Thomas http://tttwrites.in FOSS@Amrita http://foss.amrita.ac.in
*"where there is a wifi, there is a way"*
On Tue, May 27, 2014 at 9:27 PM, Bryan Davis bd808@wikimedia.org wrote:
On Tue, May 27, 2014 at 2:56 AM, Tony Thomas 01tonythomas@gmail.com wrote:
Hi, We were having discussions regarding putting the new SwiftMailer[1] lib in/out of core, after the merge of the change https://gerrit.wikimedia.org/r/#/c/135290. Tyler recommends to add the installer code to the composer.json and not to add the SwiftMailer code
to
the core. This creates the swiftmailer lib in core/vendors/swiftmailer. After discussing with Bryan (https://dpaste.de/XVkL/raw), it looks like *maintaining a separate repo* for external libraries looked like the best solution, so that it uses composer 'properly', and still works for wmf-deployement -- rather than getting the whole into core. It could be thus deployed via trebuchet to the cluster and the autoloader pulled in in CommonSetings (quoting Bryan). Since the entire mail structure is being reworded to use swiftmailer, and its a crucial dependancy ( There will be no alternate
systems in UserMailer.php -- as both the SMTP and no SMTP cases are
handled
effectively by SwiftMailer ), I think we will need to have a separate
repo
for that. Please go through the patch-set above and comment your opinions on including external libraries.
Thanks for getting this discussion started Tony. I have "been meaning to" write a very similar email regarding discussions from the Zürich hackathon about my own attempt to pull third-party libraries into mediawiki-core [2]. After talking to Ori, Tyler, Matt Walker and others in Zürich I realized that my desire to put the libraries into mediawiki-core was mostly motivated by convenience for deployment on the production wiki[mp]edia cluster. For a variety of reasons, using composer to download and install libraries directly on the production cluster is undesirable. There is an alternative however that can be clearly seen in the mediawiki/extensions/Parsoid/js/contrib.git repository.
I think we should create a new repository in gerrit ("mediawiki/core/contrib"? Bikeshed as needed) where composer is used to manage importing specific versions of external libraries that are needed for the wiki[mp]edia cluster deployments. This repository can be deployed to the production cluster using Trebuchet or scap and appropriate changes can be made in operations/mediawiki-config.git to ensure that the autoloader can find the external classes.
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Also, please note that includes/lib is meant to be a place for external libraries. Some of the libraries are ones we have ported or written ourselves, but we should continue using this space as external libraries increase in number and change in nature.
- Trevor
On Tue, May 27, 2014 at 9:12 AM, Tony Thomas 01tonythomas@gmail.com wrote:
That sounds great Bryan.
I think we should create a new repository in gerrit
("mediawiki/core/contrib"? waiting for that repo to show up in gerrit.
On Tue, May 27, 2014 at 7:38 PM, Mark A. Hershberger mah@nichework.com wrote:
Thank you! I think the more use we make of composer, the better.
I do have one concern about the tarballs we produce.
It looks like it will be necessary to pre-populate vendor with composer.json dependencies like this one -- libraries that are essential -- in order to make sure that a tarball installation require downloading anything but the tarball itself.
So, if we can have the swiftmailer repo added, this would be excellent. It might get worse otherwise, as after this shift, MW can 'only' send emails through swift. A hard dependency, as you quoted earlier.
Thanks, Tony Thomas http://tttwrites.in FOSS@Amrita http://foss.amrita.ac.in
*"where there is a wifi, there is a way"*
On Tue, May 27, 2014 at 9:27 PM, Bryan Davis bd808@wikimedia.org wrote:
On Tue, May 27, 2014 at 2:56 AM, Tony Thomas 01tonythomas@gmail.com wrote:
Hi, We were having discussions regarding putting the new SwiftMailer[1] lib in/out of core, after the merge of the change https://gerrit.wikimedia.org/r/#/c/135290. Tyler recommends to add the installer code to the composer.json and not to add the SwiftMailer code
to
the core. This creates the swiftmailer lib in core/vendors/swiftmailer. After discussing with Bryan (https://dpaste.de/XVkL/raw), it looks like *maintaining a separate repo* for external libraries looked like the best solution,
so
that it uses composer 'properly', and still works for wmf-deployement
--
rather than getting the whole into core. It could be thus deployed via trebuchet to the cluster and the autoloader pulled in in CommonSetings (quoting Bryan). Since the entire mail structure is being reworded to use swiftmailer, and its a crucial dependancy ( There will be no alternate
systems in UserMailer.php -- as both the SMTP and no SMTP cases are
handled
effectively by SwiftMailer ), I think we will need to have a separate
repo
for that. Please go through the patch-set above and comment your
opinions
on including external libraries.
Thanks for getting this discussion started Tony. I have "been meaning to" write a very similar email regarding discussions from the Zürich hackathon about my own attempt to pull third-party libraries into mediawiki-core [2]. After talking to Ori, Tyler, Matt Walker and others in Zürich I realized that my desire to put the libraries into mediawiki-core was mostly motivated by convenience for deployment on the production wiki[mp]edia cluster. For a variety of reasons, using composer to download and install libraries directly on the production cluster is undesirable. There is an alternative however that can be clearly seen in the mediawiki/extensions/Parsoid/js/contrib.git repository.
I think we should create a new repository in gerrit ("mediawiki/core/contrib"? Bikeshed as needed) where composer is used to manage importing specific versions of external libraries that are needed for the wiki[mp]edia cluster deployments. This repository can be deployed to the production cluster using Trebuchet or scap and appropriate changes can be made in operations/mediawiki-config.git to ensure that the autoloader can find the external classes.
Bryan
Bryan Davis Wikimedia Foundation bd808@wikimedia.org [[m:User:BDavis_(WMF)]] Sr Software Engineer Boise, ID USA irc: bd808 v:415.839.6885 x6855
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This may be true, but like I mentioned during the IRC RFC discussion on third-party components, it’s a case-by-case decision, and I think in this case, SwiftMailer is a fairly reliable library and definitely much better than what we have implemented now. -- Tyler Romeo 0xC86B42DF
From: Tony Thomas 01tonythomas@gmail.com Reply: Wikimedia developers wikitech-l@lists.wikimedia.org Date: May 27, 2014 at 12:13:16 To: Wikimedia developers wikitech-l@lists.wikimedia.org Subject: Re: [Wikitech-l] Adding external libraries to core (SwiftMailer)
So, if we can have the swiftmailer repo added, this would be excellent. It might get worse otherwise, as after this shift, MW can 'only' send emails through swift. A hard dependency, as you quoted earlier.
On 05/27/2014 11:57 AM, Bryan Davis wrote:
I think we should create a new repository in gerrit ("mediawiki/core/contrib"? Bikeshed as needed) where composer is used to manage importing specific versions of external libraries that are needed for the wiki[mp]edia cluster deployments.
Is the idea that third-party users would use Composer to install these libraries?
Matt Flaschen
Yes. we will have the third party code kept in a separate repo though. Currently, mw users have to run :
sudo apt-get install php-pear sudo pear install mail sudo pear install Net_SMTP
to install the pear and mail package to get mail working. After this patch, and composer loded with the swiftmailer configs as per (https://gerrit.wikimedia.org/r/#/c/135290/10/composer.json), the user just have to give
php composer.phar install
from core. and Swift gets auto loaded.
Thanks, Tony Thomas FOSS@Amrita
"where there is a wifi, there is a way"
On Wed, May 28, 2014 at 10:43 AM, Matthew Flaschen mflaschen@wikimedia.org wrote:
On 05/27/2014 11:57 AM, Bryan Davis wrote:
I think we should create a new repository in gerrit ("mediawiki/core/contrib"? Bikeshed as needed) where composer is used to manage importing specific versions of external libraries that are needed for the wiki[mp]edia cluster deployments.
Is the idea that third-party users would use Composer to install these libraries?
Matt Flaschen
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
We will have to make Jenkins bot run composer install before running tests for a start. Currently, jenkins wont be able to find the libraries and the test fails ref: https://gerrit.wikimedia.org/r/#/c/135290. Thanks, Tony Thomas FOSS@Amrita
"where there is a wifi, there is a way"
On Wed, May 28, 2014 at 7:37 PM, Tony Thomas 01tonythomas@gmail.com wrote:
Yes. we will have the third party code kept in a separate repo though. Currently, mw users have to run :
sudo apt-get install php-pear sudo pear install mail sudo pear install Net_SMTP
to install the pear and mail package to get mail working. After this patch, and composer loded with the swiftmailer configs as per (https://gerrit.wikimedia.org/r/#/c/135290/10/composer.json), the user just have to give
php composer.phar install
from core. and Swift gets auto loaded.
Thanks, Tony Thomas FOSS@Amrita
"where there is a wifi, there is a way"
On Wed, May 28, 2014 at 10:43 AM, Matthew Flaschen mflaschen@wikimedia.org wrote:
On 05/27/2014 11:57 AM, Bryan Davis wrote:
I think we should create a new repository in gerrit ("mediawiki/core/contrib"? Bikeshed as needed) where composer is used to manage importing specific versions of external libraries that are needed for the wiki[mp]edia cluster deployments.
Is the idea that third-party users would use Composer to install these libraries?
Matt Flaschen
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org