Just some updates for those not following the logs or IRC channel closely:

Images have been very slow during peak hours as load has continued to raise to a level where the disk array on the backend server can no longer keep up with peak demand. Several measures have been put in place over the week to alleviate this a bit:

brion: The image-based 'captcha' on some sites has been temporarily replaced with a text-based one until the proper captcha is updated to search its files in a more efficient way

mark: The proxy servers in front of upload.wikimedia.org have been tweaked to cache more aggressively

JeLuF: A portion of thumbnails for commons are now served through an alternate server (bacon) instead of the central one (amane)

and probably a few other tweaks.

New cache and fileserver boxes are on order and should hopefully arrive soon (some have arrived already, I think, and should be installed shortly), which will let us share the load more evenly.

There has also been some downtime to Europe due to routing problems which intermittently shut off access to our Amsterdam cluster for a few minutes while the routers scream at each other. There's not a lot we can do directly about this other than hoping that the Surfnet issues get worked out soon; it's upstream of us.

-- brion vibber (brion @ pobox.com / brion @ wikimedia.org)