I wonder if it would be possible to define some criteria that would trigger a hardware upgrade before the bottleneck becomes severe. I'm no expert on this, but I'm thinking of average response times rising above a threshold for a reasonably long period (long period in order to filter out temporary setbacks).
I am 100% in favor of this.
Here's my usual line: we have a responsibility to be good stewards of donor money, which means that we don't buy things just for the sake of buying them, and we don't buy too early... it's smart to wait for prices to go down, as they always do. At the same time, though, we donors didn't give us the money to sit on it, they want us to make wikipedia fast, and that's what we should do.
We have 4 new 1U machines on the way, to be used as apaches and squids, according to the needs identified by the developers, especially the most active in daily system management. We also have the new bigger faster DB server on the way, so that we will have 2 big fast db servers.
But we should try to estimate now (a) how long those will do and (b) what we should do in terms of the "next phase" transition.
Also, we're in need of stepping up fund raising efforts, and we want to do that *before* we have a dire need.
Right now, we have roughly $12,000 left over, although $9000+ of that is due from Penguin. That's a done deal, except I haven't gotten a check yet. :-(
So, that's enough for another small round of 1U CPU machines, which is where I think the next bottleneck will be.
--Jimbo