It looks like the problem started 3/21, see attached image.  I'm about to head to the airport for ~20 hours of travel so can't investigate further, but may be related to parser changes made that day which line up with the second bigger spike.
There was also a the release of a majorly changed MobileFrontend earlier that day / prior night that line up with the first spike.  The MF rewrite doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact non-mobile performance.

DB, ES, and memcache latency look steady, so I suspect it's all application side.


On Fri, Mar 23, 2012 at 10:00 PM, MZMcBride <> wrote:

There are over a dozen reports of general slowness with the English

As I browsed and edited the site today, I experienced similar issues
(painful slowness on intermittent page loads, loading half a page and then
stalling, etc.).

Is anyone from ops or engineering looking into this? I've filed a bug about
this as well: <>.

It may be that this is a known issue (due to schema changes or whatever
else), but I don't think anyone from ops or engineering has commented on the
village pump (or elsewhere?). Can someone please take a look at what's going


Wikitech-l mailing list