Browne is mysteriously down for the moment. An initial reboot got it running again; Tim has some syslog bits he could probably post here about some sort of problem. Apparently it went down again shortly thereafter, and my attempt to power cycle it didn't get it back, at least not back on the network.
Coronelli is currently serving all wikis. I also found that coro's squid was using an awful lot of memory - resident size ~2.1gb, though it's theoretically set to a 1350mb memory cache usage, and about 890mb used swap and growing fast. Load was 18-20ish. I restarted the squid to clear out the memory, and now the swap's gone and load's down to 6-8.
It remains to be seen whether it will eventually start eating into swap again.
The ganglia graph for coro shows slow, steady increase of swap usage and decrease of space used for cache and buffers over the last week, then vastly accelerating swap usage around the time we swapped in browne's stuff. I don't know if this indicates a memory leak somewhere or if it's supposed to be doing that.
-- brion vibber (brion @ pobox.com)