The site was randomly slow for the last week or two, maybe 10% of requests
would time out. And for a couple of days, requests through the Amsterdam
squids were timing out very regularly. Both problems are fixed now. I
apologise that it took us so long to figure out -- it was a very frustrating
problem that took a while despite the best efforts of the sysadmin team. In
both cases it was subtle LVS misconfiguration, we're developing automated
tools to test for similar problems occurring in the future.
After that was fixed, I finished setting up the 20 new dual opterons, and
put them into apache service. So service times for CPU-intensive operations
should be quite good. And of course Brion put amane into service last week,
so the problem with image load times should be fixed. The Florida squids
might still be a bit slow, we have more on order.
-- Tim Starling