Several of the cache boxen are running a bit slow; there's about a 100:1 ratio between the fast and slow boxen:

[brion@srv27 php-1.5]$ php maintenance/mctest.php 10.0.2.51:11000 set: 100 incr: 100 get: 100 time: 5.22103786469 10.0.2.55:11000 set: 100 incr: 100 get: 100 time: 0.0687239170074 10.0.2.53:11000 set: 100 incr: 100 get: 100 time: 0.0517039299011 10.0.2.54:11000 set: 100 incr: 100 get: 100 time: 0.0475029945374 10.0.2.56:11000 set: 100 incr: 100 get: 100 time: 0.0863571166992 10.0.2.70:11000 set: 100 incr: 100 get: 100 time: 0.0541288852692 10.0.2.58:11000 set: 100 incr: 100 get: 100 time: 3.91460299492 10.0.2.59:11000 set: 100 incr: 100 get: 100 time: 4.73195290565 10.0.2.59:11000 set: 100 incr: 100 get: 100 time: 4.39382100105 10.0.2.62:11000 set: 100 incr: 100 get: 100 time: 3.81929206848 10.0.2.64:11000 set: 100 incr: 100 get: 100 time: 3.57977700233 10.0.2.65:11000 set: 100 incr: 100 get: 100 time: 4.70120000839 10.0.2.66:11000 set: 100 incr: 100 get: 100 time: 0.070246219635 10.0.2.68:11000 set: 100 incr: 100 get: 100 time: 6.85797095299 10.0.2.69:11000 set: 100 incr: 100 get: 100 time: 4.67265796661 localhost set: 100 incr: 0 get: 0 time: 0.0264229774475

Any ideas?

I've also had to take srv60 out of rotation as it was timing out after insane amounts of time, hence the doubt srv59.

-- brion vibber (brion @ pobox.com)