English Wikipedia slow to load

List overview All Threads
Download

newer

older

Re: [Wikitech-l] Dropping support...

migrating a non-WMF extension to...

MZMcBride

24 Mar 2012 24 Mar '12

5 a.m.

Hi. There are over a dozen reports of general slowness with the English Wikipedia: https://en.wikipedia.org/w/index.php?oldid=483631204#is_it_me_or_is_wiki_ver y_slow.3F As I browsed and edited the site today, I experienced similar issues (painful slowness on intermittent page loads, loading half a page and then stalling, etc.). Is anyone from ops or engineering looking into this? I've filed a bug about this as well: <https://bugzilla.wikimedia.org/show_bug.cgi?id=35448>. It may be that this is a known issue (due to schema changes or whatever else), but I don't think anyone from ops or engineering has commented on the village pump (or elsewhere?). Can someone please take a look at what's going on? MZMcBride

Show replies by date

Asher Feldman

24 Mar 24 Mar

4:16 p.m.

It looks like the problem started 3/21, see attached image. I'm about to head to the airport for ~20 hours of travel so can't investigate further, but may be related to parser changes made that day which line up with the second bigger spike. - 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/CoreParserFunctions.php - 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/Parser.php - 05:23 logmsgbot: tstarling synchronized php-1.19/includes/parser/StripState.php - 05:22 logmsgbot: tstarling synchronized php-1.19/tests/parser/parserTests.txt There was also a the release of a majorly changed MobileFrontend earlier that day / prior night that line up with the first spike. The MF rewrite doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact non-mobile performance. DB, ES, and memcache latency look steady, so I suspect it's all application side. -Asher On Fri, Mar 23, 2012 at 10:00 PM, MZMcBride <z(a)mzmcbride.com> wrote:

...

Max Semenik

7:15 p.m.

On 24.03.2012, 20:16 Asher wrote:

...

There was also a the release of a majorly changed MobileFrontend earlier that day / prior night that line up with the first spike. The MF rewrite doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact non-mobile performance.

...

DB, ES, and memcache latency look steady, so I suspect it's all application side.

Thanks for the heads up, Asher. I've committed thorough profiling in <https://gerrit.wikimedia.org/r/3696>, would appreciate if someone deployed it. -- Best regards, Max Semenik ([[User:MaxSem]])

Patrick Reilly

7:33 p.m.

On Sat, Mar 24, 2012 at 12:15 PM, Max Semenik <maxsem.wiki(a)gmail.com> wrote:

...

On 24.03.2012, 20:16 Asher wrote:

DB, ES, and memcache latency look steady, so I suspect it's all application side.

Thanks for the heads up, Asher. I've committed thorough profiling in <https://gerrit.wikimedia.org/r/3696>, would appreciate if someone deployed it.

I've reviewed and merged your profiling code. We need to create a patch against the deployed version at: https://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.19wmf1/extensions… Once, that patch is created I'll apply it and push it live to production. This should give us better insight into why this slowdown in performance is occurring.

...

-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Patrick Reilly

7:51 p.m.

On Sat, Mar 24, 2012 at 12:33 PM, Patrick Reilly <preilly(a)wikimedia.org> wrote:

...

On Sat, Mar 24, 2012 at 12:15 PM, Max Semenik <maxsem.wiki(a)gmail.com> wrote:

On 24.03.2012, 20:16 Asher wrote:

DB, ES, and memcache latency look steady, so I suspect it's all application side.

Thanks for the heads up, Asher. I've committed thorough profiling in <https://gerrit.wikimedia.org/r/3696>, would appreciate if someone deployed it.

Okay, this revision: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/114477 was pushed live at 12:46pm PDT. So, we should be able to see a clear graph in about 30 minutes.

...

> > > -- > Best regards, > Max Semenik ([[User:MaxSem]]) > > > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Max Semenik

8:36 p.m.

On 24.03.2012, 23:51 Patrick wrote:

...

On Sat, Mar 24, 2012 at 12:33 PM, Patrick Reilly <preilly(a)wikimedia.org> wrote: > On Sat, Mar 24, 2012 at 12:15 PM, Max Semenik <maxsem.wiki(a)gmail.com> wrote: >> On 24.03.2012, 20:16 Asher wrote: >> >>> There was also a the release of a majorly changed MobileFrontend earlier >>> that day / prior night that line up with the first spike. The MF rewrite >>> doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to >>> 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact >>> non-mobile performance. >> >>> DB, ES, and memcache latency look steady, so I suspect it's all application >>> side. >> >> Thanks for the heads up, Asher. I've committed thorough profiling in >> <https://gerrit.wikimedia.org/r/3696>, would appreciate if someone >> deployed it. > > I've reviewed and merged your profiling code. We need to create a > patch against the deployed version at: > https://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.19wmf1/extensions… > > Once, that patch is created I'll apply it and push it live to production. > This should give us better insight into why this slowdown in > performance is occurring.

...

Okay, this revision: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/114477 was pushed live at 12:46pm PDT.

...

So, we should be able to see a clear graph in about 30 minutes.

And these graphs don't make any sense: https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse the whole function's execution time is much larger than sum of its pieces. What's going on? -- Best regards, Max Semenik ([[User:MaxSem]])

Platonides

10:28 p.m.

On 24/03/12 21:36, Max Semenik wrote:

...

And these graphs don't make any sense: https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse the whole function's execution time is much larger than sum of its pieces. What's going on?

Maybe it's a recursive function? I would have suspected that the parser fixes affected some common extension making it slower. Ganglia output shows that CPU load decreased, though. From a steady load over 10k it became much more flaky, dropping to 9k (due to full operations being slower?). Is the revision hashing still running? If there are processes fetching many old revisions from esternal storage it might be the hurting the caches (such as taking "fresh" parsed pages out of memcached). Another idea would be related to db24 failure, if some hosts failed the syncing of db.php and still pointed to it.

Max Semenik

11:22 p.m.

On 25.03.2012, 2:28 Platonides wrote:

...

On 24/03/12 21:36, Max Semenik wrote: > And these graphs don't make any sense: > https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse > the whole function's execution time is much larger than sum of its > pieces. What's going on?

...

Maybe it's a recursive function?

No, it's not - unless there's some crazy PHP quirk causing output handler to be called several times.

...

I would have suspected that the parser fixes affected some common extension making it slower. Ganglia output shows that CPU load decreased, though. From a steady load over 10k it became much more flaky, dropping to 9k (due to full operations being slower?).

...

Is the revision hashing still running? If there are processes fetching many old revisions from esternal storage it might be the hurting the caches (such as taking "fresh" parsed pages out of memcached).

Parser cache is in MySQL, you can't displace anything out of it by putting stuff into memcached.

...

Another idea would be related to db24 failure, if some hosts failed the syncing of db.php and still pointed to it.

-- Best regards, Max Semenik ([[User:MaxSem]])

Leslie Carr

25 Mar 25 Mar

1:05 a.m.

This appears to have been due to a network link issue. One of the uplinks from our row A switch in eqiad was spewing out input framing errors, causing some packets to be dropped. This link has been taken down for troubleshooting and the issue appears to be gone. Please let us know ASAP if you still are seeing any slowness. Thanks Leslie On Sat, Mar 24, 2012 at 4:22 PM, Max Semenik <maxsem.wiki(a)gmail.com> wrote:

...

On 25.03.2012, 2:28 Platonides wrote:

Maybe it's a recursive function?

No, it's not - unless there's some crazy PHP quirk causing output handler to be called several times.

Parser cache is in MySQL, you can't displace anything out of it by putting stuff into memcached.

Another idea would be related to db24 failure, if some hosts failed the syncing of db.php and still pointed to it.

-- Leslie Carr Wikimedia Foundation AS 14907, 43821

Platonides

2:59 p.m.

Max Semenik wrote:

...

Parser cache is in MySQL, you can't displace anything out of it by putting stuff into memcached.

You're wrong here. Although there's usually plenty of space, so this isn't somehting to be worried about. (I would link to a graphic of memcached servers, but I'm unable to find a ganglia entry for them)

4446

days inactive

4447

days old

wikitech-l@lists.wikimedia.org

Manage subscription

9 comments

6 participants

tags (0)

participants (6)

Asher Feldman
Leslie Carr
Max Semenik
MZMcBride
Patrick Reilly
Platonides