Hi all,
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature.
Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately.
Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM.
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era).
The API traffic is still being partially served by mod_php, but that will not be for long!
Cheers,
Giuseppe
Amazing work. Kudos for this staggered deployment! On Dec 3, 2014 6:04 PM, "Giuseppe Lavagetto" glavagetto@wikimedia.org wrote:
Hi all,
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature.
Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately.
Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM.
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era).
The API traffic is still being partially served by mod_php, but that will not be for long!
Cheers,
Giuseppe
Giuseppe Lavagetto Wikimedia Foundation - TechOps Team
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 3 December 2014 at 17:03, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature.
Excellent! Excellent! Does that make us the second-largest HHVM site?
What sort of bugs needed squashing? (Is there a list, or a suitable Phabricator query?)
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era).
How is the 1.23 tarball under HHVM? Would backports of fixes be accepted for the LTS?
- d.
On Wed, Dec 3, 2014 at 8:19 PM, David Gerard dgerard@gmail.com wrote:
What sort of bugs needed squashing? (Is there a list, or a suitable Phabricator query?)
A HHVM tag exists: https://phabricator.wikimedia.org/tag/hhvm/
This is fantastic -- kudos to everyone for pushing to get this through the finish line. Making editing faster (and improving general site responsiveness) is one of the most obvious things we can do that serves every single contributor to our projects. We've still got lots more that we can do in that area, but HHVM is a huge, huge win.
Erik
This is awesome! Great work everyone!
On Wed, Dec 3, 2014 at 9:03 AM, Giuseppe Lavagetto <glavagetto@wikimedia.org
wrote:
Hi all,
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature.
Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately.
Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM.
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era).
The API traffic is still being partially served by mod_php, but that will not be for long!
Cheers,
Giuseppe
Giuseppe Lavagetto Wikimedia Foundation - TechOps Team
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
This is fantastic. Great job team and do put up a blog post about this.
--tomasz
On Wed, Dec 3, 2014 at 9:03 AM, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
Hi all,
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature.
Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately.
Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM.
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era).
The API traffic is still being partially served by mod_php, but that will not be for long!
Cheers,
Giuseppe
Giuseppe Lavagetto Wikimedia Foundation - TechOps Team
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
Hey, I just rendered [[Barack Obama]] in about 6 seconds. I haven't tested it for several months, but in the post-Lua era I've generally gotten around 12-15 seconds. (Prior to Lua citations, it was 30 seconds.) I don't know is HHVM is the whole story, but that's a fabulous improvement towards the usability of our largest pages.
-Robert Rohde
On Wed, Dec 3, 2014 at 3:23 PM, Tomasz Finc tfinc@wikimedia.org wrote:
This is fantastic. Great job team and do put up a blog post about this.
--tomasz
On Wed, Dec 3, 2014 at 9:03 AM, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
Hi all,
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature.
Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately.
Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM.
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM
era).
The API traffic is still being partially served by mod_php, but that will not be for long!
Cheers,
Giuseppe
Giuseppe Lavagetto Wikimedia Foundation - TechOps Team
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Truly fantastic news - awesome work!
On Wed, Dec 3, 2014 at 5:01 PM, Robert Rohde rarohde@gmail.com wrote:
Hey, I just rendered [[Barack Obama]] in about 6 seconds. I haven't tested it for several months, but in the post-Lua era I've generally gotten around 12-15 seconds. (Prior to Lua citations, it was 30 seconds.) I don't know is HHVM is the whole story, but that's a fabulous improvement towards the usability of our largest pages.
-Robert Rohde
On Wed, Dec 3, 2014 at 3:23 PM, Tomasz Finc tfinc@wikimedia.org wrote:
This is fantastic. Great job team and do put up a blog post about this.
--tomasz
On Wed, Dec 3, 2014 at 9:03 AM, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
Hi all,
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who
didn't
opt-in to the beta feature.
Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous
task
that would require me weeks to complete, even with the improved automation we built lately.
Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM.
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM
era).
The API traffic is still being partially served by mod_php, but that will not be for long!
Cheers,
Giuseppe
Giuseppe Lavagetto Wikimedia Foundation - TechOps Team
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Very nice.
The impact can is also reflected and easy to spot in the "Cluster CPU" graphs on Ganglia:
https://ganglia.wikimedia.org/latest/?r=month&c=Application+servers+eqia... https://ganglia.wikimedia.org/latest/?r=year&c=Application+servers+eqiad
— Krinkle
On 3 Dec 2014, at 17:03, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
Hi all,
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature.
Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately.
Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM.
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era).
The API traffic is still being partially served by mod_php, but that will not be for long!
Cheers,
Giuseppe
Giuseppe Lavagetto Wikimedia Foundation - TechOps Team
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This deserves a 1 meter barnstar to Alex, Giuseppe and Yuvi!
The impact is so visible on * the last three graphs at https://gdash.wikimedia.org/dashboards/editpage/ and * the last four at https://gdash.wikimedia.org/dashboards/totalphp/ , that one tends to suspect an error in the graphs.
Nemo
On 03/12/14 18:03, Giuseppe Lavagetto wrote:
Hi all,
[CUT]
The API traffic is still being partially served by mod_php, but that will not be for long!
As promised, all our API traffic is on HHVM as well as of now.
The effects on CPU usage have been quite drastic on this cluster, where the load is higher:
Cheers, Giuseppe
On 12/3/14, Giuseppe Lavagetto glavagetto@wikimedia.org wrote:
Hi all,
it's been quite a journey since we started working on HHVM, and last week (November 25th) HHVM was finally introduced to all users who didn't opt-in to the beta feature.
Starting on monday, we started reinstalling all the 150 remaining servers that were running Zend's mod_php, upgrading them from Ubuntu precise to Ubuntu trusty in the process. It seemed like an enormous task that would require me weeks to complete, even with the improved automation we built lately.
Thanks to the incredible work by Yuvi and Alex, who helped me basically around the clock, today around 16:00 UTC we removed the last of the mod_php servers from our application server pool: all the non-API traffic is now being served by HHVM.
This new PHP runtime has already halved our backend latency and page save times, and it has also reduced significantly the load on our cluster (as I write this email, the average cpu load on the application servers is around 16%, while it was easily above 50% in the pre-HHVM era).
The API traffic is still being partially served by mod_php, but that will not be for long!
Cheers,
Giuseppe
Giuseppe Lavagetto Wikimedia Foundation - TechOps Team
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Awesome.
Any chance the video scalars could be put near the top of the list for servers to upgrade Ubuntu on? The really old version of libav on those servers is causing problems for people uploading videos in certain formats.
--bawolff
On 09/12/14 23:10, Brian Wolff wrote:
Awesome.
Any chance the video scalars could be put near the top of the list for servers to upgrade Ubuntu on? The really old version of libav on those servers is causing problems for people uploading videos in certain formats.
Since API and appservers are done, we're left with the jobrunners (for which the conversion is already done), the imagescalers and the videoscalers. We are working right now on the imagescaler conversion, it will require some preparation work and some testing, but it won't take too long hopefully.
Cheers,
Giuseppe
wikitech-l@lists.wikimedia.org