On 24.03.2012, 23:51 Patrick wrote:
On Sat, Mar 24, 2012 at 12:33 PM, Patrick Reilly preilly@wikimedia.org wrote:
On Sat, Mar 24, 2012 at 12:15 PM, Max Semenik maxsem.wiki@gmail.com wrote:
On 24.03.2012, 20:16 Asher wrote:
There was also a the release of a majorly changed MobileFrontend earlier that day / prior night that line up with the first spike. The MF rewrite doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact non-mobile performance.
DB, ES, and memcache latency look steady, so I suspect it's all application side.
Thanks for the heads up, Asher. I've committed thorough profiling in https://gerrit.wikimedia.org/r/3696, would appreciate if someone deployed it.
I've reviewed and merged your profiling code. We need to create a patch against the deployed version at: https://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.19wmf1/extensions/...
Once, that patch is created I'll apply it and push it live to production. This should give us better insight into why this slowdown in performance is occurring.
Okay, this revision: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/114477 was pushed live at 12:46pm PDT.
So, we should be able to see a clear graph in about 30 minutes.
And these graphs don't make any sense: https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse the whole function's execution time is much larger than sum of its pieces. What's going on?