Thank goodness new servers are forthcoming. But actually they were needed a month ago already. Edit counts have been dropping on most Wikipedias for some time now.
So far it seems we have been ordering servers when performance deteriorated. Given the fact that from ordering till installation some weeks pass:
I wonder if it would be possible to define some criteria that would trigger a hardware upgrade before the bottleneck becomes severe. I'm no expert on this, but I'm thinking of average response times rising above a threshold for a reasonably long period (long period in order to filter out temporary setbacks).
Erik Zachte
I wonder if it would be possible to define some criteria that would trigger a hardware upgrade before the bottleneck becomes severe. I'm no expert on this, but I'm thinking of average response times rising above a threshold for a reasonably long period (long period in order to filter out temporary setbacks).
I am 100% in favor of this.
Here's my usual line: we have a responsibility to be good stewards of donor money, which means that we don't buy things just for the sake of buying them, and we don't buy too early... it's smart to wait for prices to go down, as they always do. At the same time, though, we donors didn't give us the money to sit on it, they want us to make wikipedia fast, and that's what we should do.
We have 4 new 1U machines on the way, to be used as apaches and squids, according to the needs identified by the developers, especially the most active in daily system management. We also have the new bigger faster DB server on the way, so that we will have 2 big fast db servers.
But we should try to estimate now (a) how long those will do and (b) what we should do in terms of the "next phase" transition.
Also, we're in need of stepping up fund raising efforts, and we want to do that *before* we have a dire need.
Right now, we have roughly $12,000 left over, although $9000+ of that is due from Penguin. That's a done deal, except I haven't gotten a check yet. :-(
So, that's enough for another small round of 1U CPU machines, which is where I think the next bottleneck will be.
--Jimbo
On Tue, 18 May 2004 09:05:43 -0700 Jimmy Wales jwales@bomis.com wrote:
Right now, we have roughly $12,000 left over, although $9000+ of that is due from Penguin. That's a done deal, except I haven't gotten a check yet. :-(
Could you update http://wikimediafoundation.org/fundraising ? Some people will give money if there's litle in bank :)
So, that's enough for another small round of 1U CPU machines, which is where I think the next bottleneck will be.
And if we want to get fast Suda, we'll need to upgrade his system disk. 2 more 146 GB drive so we would set a raid 10 with 1 spare drive.
Shaihulud
Jimmy, I know this has been brought up before, but have you called up brad from live-journal? I follow Live journal's tech stuff pretty closely and once again I really think you could benefit from having them help you plan something out. After all they are experts at this. These guys have a couple of racks full of machines and order servers more often than most people order pizza. Not only that but they have something very similar to what wikipedia has, a system that is incredibly DB intensive (they use mysql too) with huge numbers of writes occurring every second of the day. They also cater to people around the world, have had to deal with i18n. They developed memcached when mysql replication couldn't do it for them anymore, and have developed some incredibly redundant systems using hardware very similar to wikipedias (oh, and they buy from SM too) etc. I am sure they would be glad to say something, or give you a couple of tips here or there on the topic of managing all these machines, creating a redundant system, and squeezing out the most efficiency possible out of what wikipedia has. Again, just a thought.. oh and note to Brion and other devs: They also might have some tips and pointers related to apache/mysql tuning and memcacheD. If this is too late.. well then that's good, but if this hasn't happened, I am confident the staff over at LJ wouldn't mind a chat with you guys, they are probably big fans of the project.
Oh one final not. I dont know how images and files are being served up currently, but I really think a server dedicated to serving them would be great, especially one with a really small http server, no frills no thrills, tiny memory print, etc. It might help to serve images like that instead of through apache processes which are bigger, heavier and more demanding. once again, if this hasn't been done yet I'm sure you guys are either busy with other stuff or there's a good reason behind it.. but just in case
Lightning
On May 18, 2004, at 12:05 PM, Jimmy Wales wrote:
I wonder if it would be possible to define some criteria that would trigger a hardware upgrade before the bottleneck becomes severe. I'm no expert on this, but I'm thinking of average response times rising above a threshold for a reasonably long period (long period in order to filter out temporary setbacks).
I am 100% in favor of this.
Here's my usual line: we have a responsibility to be good stewards of donor money, which means that we don't buy things just for the sake of buying them, and we don't buy too early... it's smart to wait for prices to go down, as they always do. At the same time, though, we donors didn't give us the money to sit on it, they want us to make wikipedia fast, and that's what we should do.
We have 4 new 1U machines on the way, to be used as apaches and squids, according to the needs identified by the developers, especially the most active in daily system management. We also have the new bigger faster DB server on the way, so that we will have 2 big fast db servers.
But we should try to estimate now (a) how long those will do and (b) what we should do in terms of the "next phase" transition.
Also, we're in need of stepping up fund raising efforts, and we want to do that *before* we have a dire need.
Right now, we have roughly $12,000 left over, although $9000+ of that is due from Penguin. That's a done deal, except I haven't gotten a check yet. :-(
So, that's enough for another small round of 1U CPU machines, which is where I think the next bottleneck will be.
--Jimbo
Wikitech-l mailing list Wikitech-l@Wikipedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l
On Tue, 18 May 2004 13:05:40 -0400 Lightning lightning@creativechemistry.net wrote:
Oh one final not. I dont know how images and files are being served up currently, but I really think a server dedicated to serving them would be great, especially one with a really small http server, no frills no thrills, tiny memory print, etc. It might help to serve images like that instead of through apache processes which are bigger, heavier and more demanding. once again, if this hasn't been done yet I'm sure you guys are either busy with other stuff or there's a good reason behind it.. but just in case
We have a nfs server for the pictures and some php files. But now it is not well configured. I think we'll be able to upgrade it with a raid1 setup when Jimmy will install the new boxes.
Shaihulud
wikitech-l@lists.wikimedia.org