Just some updates for those not following the logs or IRC channel closely:
Images have been very slow during peak hours as load has continued to raise to a
level where the disk array on the backend server can no longer keep up with peak
demand. Several measures have been put in place over the week to alleviate this
a bit:
brion: The image-based 'captcha' on some sites has been temporarily replaced
with a text-based one until the proper captcha is updated to search its files in
a more efficient way
mark: The proxy servers in front of
upload.wikimedia.org have been tweaked to
cache more aggressively
JeLuF: A portion of thumbnails for commons are now served through an alternate
server (bacon) instead of the central one (amane)
and probably a few other tweaks.
New cache and fileserver boxes are on order and should hopefully arrive soon
(some have arrived already, I think, and should be installed shortly), which
will let us share the load more evenly.
There has also been some downtime to Europe due to routing problems which
intermittently shut off access to our Amsterdam cluster for a few minutes while
the routers scream at each other. There's not a lot we can do directly about
this other than hoping that the Surfnet issues get worked out soon; it's
upstream of us.
-- brion vibber (brion @
pobox.com / brion @
wikimedia.org)