On 11/06/14 03:24, John wrote: Why? This sounds like a really bad idea, but maybe it isn't, and as a sysadmin and developer I need to know what it even is so I can decide if I should flip the fuck out. Or not. -I

On 11/06/14 03:31, John wrote: Okay, so I'm completely lost. Is the idea here that SwiftMailer is some shiny new thing that introduces new features, but also new dependencies and whatnot, and that MediaWiki as a whole won't work without the dependencies involved? If so, does it really need to be in core? And if it doesn't, why isn't it an extension? And even if it does need to be in core, why should it mean MediaWiki should no longer be installable from git? MediaWiki already has dependencies (which are generally met for all wikis when several are already installed), and the installation process either allows you to progress or not depending on whether or not they are met, but it does at least start out of the box. Why should this be any different? Is this really a sane transition/requirement for developers running test instances, for third-party projects running from git, and other users? And, again, what's the purpose of it in the first place? -L

On 2014-06-10, 8:59 PM, John wrote: Sorry but these strawmen are quite annoying, and frankly disrespectful to the developers trying to improve core. SwiftMailer is not some toy or shiny trinket, it is a serious and well maintained library dedicated to sending email from PHP. Adding it to core is not some new feature that would be better in an extension. It is a serious improvement to our core handling of email within PHP that replaces our crude UserMailer code. Our current UserMailer code is ridiculous when you look at it. By default our UserMailer code will just use php's creaky mail() function. Which has some 'features' like: - Unpredictable message headers - Lack of feedback regarding delivery failures - And this beautiful comment in our codebase: # PHP's mail() implementation under Windows is somewhat shite, and # can't handle "Joe Bloggs <joe(a)bloggs.com>" format email addresses, # so don't bother generating them If you want to send email a different way, you could of course instead use $wgSMTP which: - First requires you to install PEAR Mail. - Who the hell uses PEAR anymore! - And don't forget, PEAR is a tool that installs modules globally and is difficult to impossible to use without shell and even admin access. - ;) And this is the hell we put tarball users through, not devs. - This PEAR Mail library we're relying on to handle all users who can't use mail() or don't want it's features, here's the dev page: - http://pear.php.net/package/Mail - Current stable 1.2.0, released in March of 2010 - Next release, 1.2.1 scheduled release date in January of 2011, never released. - ((In other words, the library we're entrusting all our SMTP handling to is practically dead and no longer maintained, Whoops.)) - Admittedly if you hunt down PEAR Mail's github repo there has been a bit of activity: - https://github.com/pear/Mail/commits/trunk - But none of this will be installed by pear, it'll only install old code from 2010 missing fixes for a number of bugs in PEAR Mail - The majority of changes in 2014 have been minor/trivial things (adding travis builds, whitespace fixes) - And, ;) making PEAR Mail installable via Composer *snicker* And to sprinkle this all off, because mail() and PEAR Mail are so different, half the code in UserMailer::send() is split into two different large code paths, which is a recipe for maintenance headaches. Using Swift Mailer on the other hand: - The library has ample development and maintenance going on: https://github.com/swiftmailer/swiftmailer/commits/master - SMTP and mail() are abstracted away into transports, so instead of two large (unmaintained?) code paths we just craft and send an email in one code path, a well maintained library takes care of the difference between transports. - In the future we can move away from using mail() and add the ability to use sendmail as a transport directly, without the bugs (in theory I think it would even be possible to try swapping a different transport in place of mail() automatically). - All this gets bundled into the tarball directly, so $wgSMTP now works out of the box and doesn't require installation of something that in some situations is impossible to install. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]

On 2014-06-11, 12:07 AM, Tim Starling wrote: Well yes, there are only a few (actually 4 not 3) open bugs: http://pear.php.net/bugs/search.php?cmd=display&package_name[]=Mail&… Although there's also this: http://pear.php.net/bugs/roadmap.php?package=Mail&roadmapdetail=1.2.1#a… Because 1.2.1 was never released to pear there are 6 extra already fixed bugs which everyone using PEAR Mail are still affected by. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]

On 2014-06-11, 10:55 PM, Tim Starling wrote: Oh no, that's not what the snicker is about. I was replying to an email completely against the introduction of composer and Swift Mailer to replace the current core UserMailer status quo. ;) So I was snickering at the irony that one of the few changes made to the library we currently use, is support for the very installer that the email I was replying to was opposing. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/]

On 11/06/14 13:22, Isarra Yos wrote: In the current change, the purpose is to provide a clean, feature-rich interface which developers can use to send emails. In particular, there is a mail library called SwiftMailer which provides bounce detection, among other things. Bounce detection would be a nice thing to have. It is proposed that developers use SwiftMailer directly instead of using a MediaWiki wrapper which optionally uses SwiftMailer. This would make the MW core smaller and thus easier to maintain. -- Tim Starling

On 11.06.2014 9:01, Matthew Walker wrote: PSR loader is ok. PSR "standard" of using four spaces instead of tabs for indentation is strange. That prevents from easily adjusting the indentation (good editors can visually render tabs like N-spaces according to user preference) and has another drawbacks. composer can use git to fetch dependencies, at least it does so when I developed with symfony2. https://getcomposer.org/doc/05-repositories.md Dmitriy

On Wed, Jun 11, 2014 at 7:20 PM, Isarra Yos <zhorishna(a)gmail.com> wrote: Which use cases are you referring to exactly? (I'm curious as to how composer does not work with farms.) *-- * *Tyler Romeo* Stevens Institute of Technology, Class of 2016 Major in Computer Science

On 11/06/14 14:49, I wrote: Sorry, it seems I was misled on this. SwiftMailer does not have any special handling for bounces, judging by the code review I have done just now and <http://pookey.co.uk/wordpress/archives/183-using-verp-with-swiftmailer-symfony-and-exim> All it has is a setReturnPath() method, and of course we have equivalent code in UserMailer.php already: $headers['Return-Path'] = $from->address; With VERP this would become: $headers['Return-Path'] = $this->makeReturnPath( $from ); or some such. With SwiftMailer, you could instead have: $message->setReturnPath( $this->makeReturnPath( $from ) ); ...with identical implementation of makeReturnPath(). SwiftMailer does not give you any help in receiving bounces. It has no MIME parsers or clients for POP, IMAP, MTA pipes, etc. On [[Talk:VERP]], Tyler suggested writing a SwiftMailer plugin to "implement VERP", by which he apparently means a plugin which would serialize the relevant parameters and calculate an HMAC, and then set the return path. Presumably this would look like: $verp = new Swift_Plugins_VerpPlugin( array( 'wiki' => wfWikiId(), 'ua' => 'UserMailer', 'time' => wfTimestampNow() ), $secretKey ) ); $mailer->registerPlugin( $verp ); $mailer->send( $message ); With Swift_Plugins_VerpPlugin being along the lines of: class Swift_Plugins_VerpPlugin { function __construct( $context ) { $this->context = $context; } function beforeSendPerformed( $evt ) { $message = $evt->getMessage(); $message->setReturnPath( $this->makeReturnPath( $message->getReturnPath() ) ); } ... } ...with makeReturnPath() being the serialization code, equivalent to my suggestion of UserMailer::makeReturnPath() above. On [[Talk:VERP]], Jeff Green was fairly skeptical about the value of doing it this way, and I guess I am too. You have to wonder where the boundary is between abstraction and needless complexity. And where do you put the receive code? Surely the return path parser and the return path printer should be maintained together. SwiftMailer is basically just an SMTP client. I would think that a bounce processing feature as a whole would be an elegant thing to put a module boundary around, rather than having half of it implemented as an SMTP client plugin and the other half implemented as a MediaWiki-specific CLI interface for an MTA pipe. In other news, I found a serious security vulnerability in SwiftMailer and have reported it upstream. -- Tim Starling

What about doing the reasonable thing and leaving core the hell alone? this should be an extension and not shoved into core. On Tue, Jun 10, 2014 at 11:43 PM, Daniel Friesen &lt;daniel(a)nadir-seen-fire.com

Just to add to the technical stuff:- * We made a local repo to fork the original swiftmailer repo at https://github.com/wikimedia/mediawiki-core-vendor-swiftmailer/tree/5.2.0-p…, and our https://gerrit.wikimedia.org/r/#/c/137538/ makes composer take up the files from our repo and not the upstream one, which is for the development end. * Our VERP project is going on at https://gerrit.wikimedia.org/r/#/c/138655/ Thanks, Tony Thomas <http://tttwrites.in> FOSS@Amrita <http://foss.amrita.ac.in> *"where there is a wifi, there is a way"* On Wed, Jun 11, 2014 at 1:10 PM, Adrian Lang &lt;adrian.lang(a)wikimedia.de&gt; wrote:

On Wed, Jun 11, 2014 at 1:58 PM, Antoine Musso &lt;hashar+wmf(a)free.fr&gt; wrote: Please find Bryan's RFC here:- https://www.mediawiki.org/wiki/Requests_for_comment/Composer_managed_librar… Thanks, Tony Thomas <http://tttwrites.in> FOSS@Amrita <http://foss.amrita.ac.in> *"where there is a wifi, there is a way"*

OK, now that I have some more time, I will expand on what I’ve already said in the Gerrit patch. == Introduction == The WMF does not have that many developers. And it’s not that they haven’t hired enough or anything like that. It’s just they do not have that many people in general. As of last year, they had under 200 employees, total. I work at an enterprise software company, and we have 200 developers alone on the app/dev team, let alone the additional people (including myself) in the architecture group, in the business analyst team, in management, etc. To think that under 200 people can develop a 400,000+ line codebase is insane. That’s why the WMF gets help from the open source community. Not only because people love to work on open source projects that benefit the world and humanity, but because such projects *need* volunteers because they simply do not have the billion-dollar operating revenue necessary to develop an enterprise product. With that said, it is absolutely necessary that the WMF get as much help as it can. Right now it has a lot of help: numerous volunteer developers working on core, working on translatewiki, etc. However, it’s not easy convincing people to join an open source software project. I’m sure Quim and Sumana can attest to this. What we can do, however, is “export” the work. == Third-party libraries == === How they can help? === Third-party libraries, i.e., other open-source software projects, are helpful in that: 1) They have existing communities of interested volunteer (and sometimes even paid) developers. 2) They are structured in an abstract manner, meaning they can be used inside MediaWiki without having extensive knowledge of the library’s internals. 3) They help to remove possibly hundreds if not thousands of lines of code from MediaWiki core that does not need to be there. The premise is: why reinvent the wheel? Or, to be more specific, why reinvent the wheel by carving a square stone rock? There are libraries out there (such as SwiftMailer, which I’ll get to) that do certain tasks that MediaWiki needs to do. However, they are ten times as comprehensive as MediaWiki’s make-shift implementations, and are maintained a lot more. === The qualifications === Obviously, we cannot just start throwing in third-party libraries willy-nilly. There are major implications, specifically: * Possible security issues introduced by the library * Having to submit patches upstream when bugs are discovered These are real issues. However, if the third-party library we are thinking of including is: 1) popular, i.e., used by other projects, especially commercial and/or commercially-backed projects; 2) uses a sane and reliable code review process and release process; and 3) is stable and unlikely to change drastically, then the library is reliable, and we do not really have to worry about issues. One argument that people seem to be making is that since the code did not go through *our* review process, the code might be vulnerable or insecure. That is a fallacy. To say that code going through our own review process is more secure than others is ridiculous. Doing a security review of code requires security knowledge, and those with security knowledge (i.e., Chris) are not reviewing every line of code that goes into core. Long story short, we do need to evaluate what libraries we put in, but if they are reliable it will not be a problem. == SwiftMailer == Now we get to SwiftMailer. Some are calling it a new shiny toy, and like Daniel said, this is a bit insulting to anybody who has even looked at the UserMailer.php code. With SwiftMailer, the only mail code that will be in MediaWiki will be what is necessary to get the e-mail address of a user. Everything else is handled inside the library. We no longer have to maintain mail-sending code at all. Putting PEAR aside, considering the major annoyances in using PHP’s mail() function, including differences in handling between operating systems, it is a big success if we do not have to deal with the mail() function inside MediaWiki. --  Tyler Romeo 0xC86B42DF From: Tyler Romeo tylerromeo(a)gmail.com Reply: Tyler Romeo tylerromeo(a)gmail.com Date: June 11, 2014 at 7:21:46 To: Wikimedia developers wikitech-l(a)lists.wikimedia.org Subject:  Re: [Wikitech-l] Making a plain MW core git clone not be installable On Wed, Jun 11, 2014 at 4:28 AM, Antoine Musso &lt;hashar+wmf(a)free.fr&gt; wrote: - depends on upstream to incorporate our patches - might not be used as-is on Wikimedia cluster I would just like to point out that you can override where Composer finds packages using the composer.json file. In other words, if we wanted to force Composer to use WMF's git repositories to check out dependencies, that is entirely possible. In the end, Composer uses git to check out the packages anyway. https://getcomposer.org/doc/04-schema.md#repositories --  Tyler Romeo Stevens Institute of Technology, Class of 2016 Major in Computer Science

I'm not a developer so it's perfectly normal that I can't understand anything about your talk; nevertheless, please, remember of KISS principle when building any installing tools for poor, final users. I'm waiting for something like "pip install core". Alex 2014-06-11 15:58 GMT+02:00 C. Scott Ananian &lt;cananian(a)wikimedia.org&gt;rg>:

On Wed, Jun 11, 2014 at 10:51 AM, Tyler Romeo &lt;tylerromeo(a)gmail.com&gt; wrote: ... That's just awful.

On Wed, Jun 11, 2014 at 10:58 AM, Tyler Romeo &lt;tylerromeo(a)gmail.com&gt; wrote: Well, it makes *me* wince because you're directing people to pull code over the network and feed it straight to the PHP interpreter, probably as root, without inspecting it first. And the site is happy to send it to you via plain HTTP, which means a one-character typo gives an active attacker a chance to pwn your entire installation. No, nobody bothers to read all the code they just checked out of Git, but it's integrity-protected by design, independent of the transport channel -- you know that the code you just received is the exact same code everyone else is getting, so you can trust that *someone* did the security audit. (And yeah, no one does *that* either, which is how we got the OpenSSL fiasco, but computers can't solve that problem.) zw

Ouch, thanks for wasting a few of my brain cells. This is why do dont add stupid code to core. My web server doesnt have curl installed, nor does it have /usr/bin/local/ You havent bothered to think your code through. Why dont you un-fuck your code, configure it as an extension and go from there? at that point you can find out exactly how many site your going to break. Once you have a stable reviewed extension we can *think* about merging it to core. On Wed, Jun 11, 2014 at 11:21 AM, Tyler Romeo &lt;tylerromeo(a)gmail.com&gt; wrote:

On Wed, Jun 11, 2014 at 11:21 AM, Tyler Romeo &lt;tylerromeo(a)gmail.com&gt; wrote: Nothing stops you from installing it over insecure HTTP. (I filed https://github.com/composer/composer/issues/3047 for that.) But this is bad practice even with HTTPS; you're relying on *transport* integrity/authenticity to secure *document* authenticity. Yeah, we do that all the time on today's Web, but software installation is (I don't think this is hyperbole) more security-critical than anything else and should be held to higher standards. In this case, there should be an independently verifiable (i.e. not tied to the TLS PKI) PGP signature on the installer and people should be instructed to check that before executing it. Note that Git submodules do this for you automatically, because the revision hash is unforgeable. zw

Thanks Zack for actually explaining the reasoning to me, rather than trying to insult my intelligence and then use it as an argument against the proposal. --  Tyler Romeo 0xC86B42DF From: Zack Weinberg zackw(a)cmu.edu Reply: Wikimedia developers wikitech-l(a)lists.wikimedia.org Date: June 11, 2014 at 11:47:34 To: Wikimedia developers wikitech-l(a)lists.wikimedia.org Subject:  Re: [Wikitech-l] Making a plain MW core git clone not be installable Nothing stops you from installing it over insecure HTTP. (I filed https://github.com/composer/composer/issues/3047 for that.) But this is bad practice even with HTTPS; you're relying on *transport* integrity/authenticity to secure *document* authenticity. Yeah, we do that all the time on today's Web, but software installation is (I don't think this is hyperbole) more security-critical than anything else and should be held to higher standards. In this case, there should be an independently verifiable (i.e. not tied to the TLS PKI) PGP signature on the installer and people should be instructed to check that before executing it. Note that Git submodules do this for you automatically, because the revision hash is unforgeable.

On Wed, Jun 11, 2014 at 4:28 AM, Antoine Musso &lt;hashar+wmf(a)free.fr&gt; wrote: Couldn't you do a shallow clone, if the disk space or bandwidth is a concern? This could well be a minus if it winds up being done poorly. + could be used to handle extensions dependencies Well, we could fork if necessary. Although that's seldom fun. This is one of my pet peeves. It's one thing to have to install dependencies to run MediaWiki, but having to install various dependencies just to *install* it? Ugh! Composer is a first step. Then there's the proposals floating around to make various nodejs services be required (which themselves tend to be a pain because they seem to require dozens of npm packages), which leads to proposals that MediaWiki require puppet/docker/something like that just to be installed because of the overly complex interdependencies. I have nothing against puppet/docker/etc in general, but it seems like overkill for running a simple MediaWiki installation off of a single host. - might not be used as-is on Wikimedia cluster -- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation

Le 11/06/2014 15:51, Brad Jorsch (Anomie) a écrit : composer is being more and more used in the PHP world and I see it as essentially replacing Pear that probably nobody is going to regret. So if one is going to install MediaWiki from git, there is a very high chance he already has composer and the person would definitely be able to figure out how to install it. For end users, they would rely on tarballs that ships the third parties dependencies. The orientation toward service oriented architecture (ie bunch of well defined API implemented in whatever language) would surely make it harder and harder to keep supporting third parties. But that is another can of worm. -- Antoine "hashar" Musso

(anonymous) wrote: Shouldn't http://pear.php.net/manual/en/installation.shared.php allow installation of PEAR packages in local directories? Tim