Brion Vibber wrote:
The Florida upload squid cluster started experiencing a problem with CPU usage maxed out and slow responses at about 16:45 UTC (45 minutes ago).
There is no problem with the internal file server, but images are loading very slowly via the web due to some kind of overload on the HTTP proxies.
We're poking at it but have not yet resolved the issue.
We've disabled the IE 6 PNG workaround on en.wikipedia.org for now: http://en.wikipedia.org/wiki/MediaWiki_talk:Common.js#PNG_fix_disabled
The problem was manifesting primarily with loads of [[Image:Must left-click image again before saving.gif]], which was being substituted in for PNG images by the workaround code.
Backtraces seemed to be showing insanely long loops of httpHeaderGetEntry(), indicating possible memory corruption or other extreme weirdness.
We're not entirely sure how consistent it is or what exactly triggers it, but please don't turn it back on in the way it's currently coded without checking with us first.
-- brion vibber (brion @ wikimedia.org)