This appears to have been due to a network link issue. One of the uplinks from our row A switch in eqiad was spewing out input framing errors, causing some packets to be dropped.
This link has been taken down for troubleshooting and the issue appears to be gone. Please let us know ASAP if you still are seeing any slowness.
Thanks Leslie
On Sat, Mar 24, 2012 at 4:22 PM, Max Semenik maxsem.wiki@gmail.com wrote:
On 25.03.2012, 2:28 Platonides wrote:
On 24/03/12 21:36, Max Semenik wrote:
And these graphs don't make any sense: https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse the whole function's execution time is much larger than sum of its pieces. What's going on?
Maybe it's a recursive function?
No, it's not - unless there's some crazy PHP quirk causing output handler to be called several times.
I would have suspected that the parser fixes affected some common extension making it slower. Ganglia output shows that CPU load decreased, though. From a steady load over 10k it became much more flaky, dropping to 9k (due to full operations being slower?).
Is the revision hashing still running? If there are processes fetching many old revisions from esternal storage it might be the hurting the caches (such as taking "fresh" parsed pages out of memcached).
Parser cache is in MySQL, you can't displace anything out of it by putting stuff into memcached.
Another idea would be related to db24 failure, if some hosts failed the syncing of db.php and still pointed to it.
-- Best regards, Max Semenik ([[User:MaxSem]])
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l