Hi! This is a quick heads-up about the status of HHVM migration, and what the MediaWiki Core team is working on.
There are three challenges that we have to solve before we can run HHVM in production:
* We need good packages. The packages provided by Facebook have some deep issues that need to be fixed before they meet our packaging standards. This is a good opportunity to recognize Faidon's leadership on this front: he has been liasoning with Facebook and Debian, working to resolve the outstanding issues. Thanks, Faidon! * We need to port a bunch of C extensions to the Zend PHP interpreter to HHVM. The most complex by far is LuaSandbox. Tim has been working on that. In the process, he has made substantial improvements to the Zend extension compatibility layer provided by HHVM, which we are waiting to have merged upstream: https://github.com/facebook/hhvm/pull/1986. Once they are merged, they will be in the queue for the next release. Releases are cut every eight weeks. * I also want to recognize Max Seminik, who stepped up to port Wikidiff2, producing a patch in short order. * We need to adapt our app server configuration for HHVM. This includes configuring HHVM itself as well as reconfiguring Apache to act as a fastcgi reverse-proxy. * We need to amend our deployment process so that it implements additional requirements for HHVM. Specifically, we will need to add a build step to produce a bytecode archive in advance of deployment. We are not working on that piece yet, but I think that Bryan's work on scap is going to make this a lot easier to implement once we do tackle it.
What we've done so far is to use Facebook's packages in Labs and in MediaWiki-Vagrant, configured Jenkins to run the unit tests under HHVM (Antoine), and configured a Jenkins job to build HHVM from source hourly so we can test patches (Chad). Aaron and I reasoned our way out of having to port the igbinary extension, and Aaron is now working on porting FastStringSearch. Along the way, we have been running into small compatibility nits which we have fixed either by changing core's behavior to be cross-compatible or by filing bugs and submitting patches upstream.
As you can see, there are some hard blockers that stand between us and HHVM in production, and the biggest ones are not entirely in our hands (i.e., they depend on upstream merging patches and fixing packages). At the same time, there is a lot of useful work left to do that can continue without being blocked by these things. For that reason, the Core MediaWiki team is currently targetting the Beta cluster for HHVM work.
Our target for the current sprint is to have the ability to have Apache run either the Zend interpreter or HHVM based on the presence of a magic cookie. By default, visitors to the beta cluster will be served pages generated using the Zend interpreter, but by setting the cookie, Apache would serve MediaWiki using HHVM instead. This is an idea we got from Niklas, who has implemented something very similar for < http://dev.translatewiki.net/%3E. Doing this this would allow the beta cluster to continue to be faithful to production and thus continue to be a good target for testing, while at the same time provide a way for people working on HHVM specifically to test ported extensions and to identify and fix integration points in a production-like environment. It also gives us a way of making our progress visible to you.
We have benchmarked different workloads on different hardware and have found the performance of HHVM to be impressively better than the Zend interpreter in most cases, but we don't yet have numbers to share that project the impact on users, because we don't have the means of simulating the load patterns of production, and because some parts of the stack are still in the process of being ported. We expect that having the option of running HHVM on the Beta cluster with the complete set of extensions that Wikimedia uses will make it possible for us to project how it will perform in production. But we are optimistic, given what we've observed and given the spate of independent evaluations of HHVM from different corners of the PHP community.
We are using Bugzilla to track our progress. You can search for bugs with the 'hiphop' keyword, or simply head to https://www.mediawiki.org/wiki/HHVM, which aggregates the most recently touched items via RSS. If you'd like to get involved, pick an open bug, or get in touch via the lists or IRC.
Regards, Core Platform.
Thanks for the update, Ori. Exciting stuff :)
On Fri, Mar 21, 2014 at 03:42:41AM -0700, Ori Livneh wrote:
- We need good packages. The packages provided by Facebook have some deep
issues that need to be fixed before they meet our packaging standards. This is a good opportunity to recognize Faidon's leadership on this front: he has been liasoning with Facebook and Debian, working to resolve the outstanding issues. Thanks, Faidon!
To be clear, since I saw some discussion on the IRC backlog: this isn't about "Debian packaging standards", the "Debian way" or anything like that. The packages they provide are not real packages, they are essentially just tarballs packaged in the ar format and extractable with dpkg. It's in the form of this ugly, Facebook-specific, unmaintainable shell script: https://github.com/hhvm/packaging/blob/master/hhvm/deb/package
It's not "packages" that can be improved by us or anyone else. There is progress on making packages, and a Debian Developer that works at Facebook has been enlisted, on company time AIUI, to also help them with that. There is work of his & mine in progress at: http://anonscm.debian.org/gitweb/?p=collab-maint/hhvm.git;a=shortlog It's already buildable and probably better already than the dl.hhvm.com packages. I'd be perfectly okay with these packages finding their way into Wikimedia production, even *before* they are up to Debian standards and suitable for an upload into Debian proper.
There /are/ some interactions with the Debian project that are on our best interests, as they might help us foster a healthy community around HHVM, e.g. http://lists.alioth.debian.org/pipermail/pkg-php-maint/2014-January/012988.h... but these are nice-to-haves, not blockers for our HHVM production deployment.
- We need to port a bunch of C extensions to the Zend PHP interpreter to
HHVM. The most complex by far is LuaSandbox. Tim has been working on that. In the process, he has made substantial improvements to the Zend extension compatibility layer provided by HHVM, which we are waiting to have merged upstream: https://github.com/facebook/hhvm/pull/1986. Once they are merged, they will be in the queue for the next release. Releases are cut every eight weeks.
- I also want to recognize Max Seminik, who stepped up to port Wikidiff2,
producing a patch in short order.
Note that there is still the outstanding issue of how to deploy these extensions, as it's my understanding that HHVM's ABI is not stable and hence they would need to be rebuilt with every new HHVM version using the HHVM source tree. It's a bit messy, Tim has all the details.
- We need to adapt our app server configuration for HHVM. This includes
configuring HHVM itself as well as reconfiguring Apache to act as a fastcgi reverse-proxy.
This will also require some puppet work -- the current classes aren't great and it won't be too easy to plug an alternative implementation without some rework. Not too much work, but have it on your radar nevertheless.
Faidon
This sounds all exiciting but let me remind you that if we want to get out of Tampa we need to move the current _Apache_ deployment. Sync scripts,-gracefull all etc away from fenari to something in eqiad. Is there anyone working on this? Just making sure that we don't fall into the usual trap - everybody wants to build new stuff - nobody wants to migrate old boring stuff.
wikitech-l@lists.wikimedia.org