Daniel Mayer wrote:
The issue as I see it, is that we are now asking too much of too few volunteer developers and none of them can be on-site at the colo. Thus I think it is the time to consider hiring one full time or two of them part time to perform server maintenance tasks for us that currently are not done in a timely manor (esp the on-site stuff). Our budget is now so big ($US80,000 last quarter) that the addition of one or two employees under contract would not be too much of a burden. Neither would some up-front relocation money.
We're doing this already, paying the guy who goes by the name "Baylink" on IRC. More servers would also be nice -- more in every category. We're having trouble using the hardware we have efficiently due to the lack of system administrators and developers, but inefficiency can be compensated for by buying more hardware.
The cheap way to get more hardware is to fix the ~7 machines that are currently broken and under warranty. Setting up new machines is a significant time cost to the system administrator team, it would be better if the existing machines were fixed and put back into service as soon as possible after breaking. The cluster configuration changes constantly, if a machine is out of action for long enough, reconfiguring it to match the rest of the cluster becomes as hard as adding a new machine.
There's not much any of us can do to get machines fixed besides hassle Jimbo.
I'm told 10 machines were ordered yesterday, that will help. Here are the details:
http://meta.wikimedia.org/wiki/Hardware_ordered_January_2005
I suspect any difference between the Paris squids and the Florida squids will disappear when those dual-H/D machines are put into service. However I don't think it will be long before the apache cluster needs attention again.
JamesDay is of the opinion that we need two more database slaves worth $10,000-$15,000 each. I'd add that to a large degree, available database hardware limits what we can consider implementing in MediaWiki. We typically live on the breadline, but we can always find a use for more database hardware.
-- Tim Starling