This appears to have been due to a network link issue. One of the
uplinks from our row A switch in eqiad was spewing out input framing
errors, causing some packets to be dropped.
This link has been taken down for troubleshooting and the issue
appears to be gone. Please let us know ASAP if you still are seeing
any slowness.
Thanks
Leslie
On Sat, Mar 24, 2012 at 4:22 PM, Max Semenik <maxsem.wiki(a)gmail.com> wrote:
On 25.03.2012, 2:28 Platonides wrote:
On 24/03/12 21:36, Max Semenik wrote:
> And these graphs don't make any sense:
>
https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse
> the whole function's execution time is much larger than sum of its
> pieces. What's going on?
Maybe it's a recursive function?
No, it's not - unless there's some crazy PHP quirk causing output
handler to be called several times.
I would have suspected that the parser fixes
affected some common
extension making it slower.
Ganglia output shows that CPU load decreased, though. From a steady load
over 10k it became much more flaky, dropping to 9k (due to full
operations being slower?).
Is the revision hashing still running? If there
are processes fetching
many old revisions from esternal storage it might be the hurting the
caches (such as taking "fresh" parsed pages out of memcached).
Parser cache is in MySQL, you can't displace anything out of it by
putting stuff into memcached.
Another idea would be related to db24 failure, if
some hosts failed the
syncing of db.php and still pointed to it.
--
Best regards,
Max Semenik ([[User:MaxSem]])
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l