On 9/3/05, Tim Starling t.starling@physics.unimelb.edu.au wrote:
Evan Martin wrote:
I believe this is correct. We've had problems in the past with overloading wikipedia so the crawl has been throttled way back.
Throttled back to what? I remember having problems in late 2003 and early 2004, but we have 30 times as much hardware now, so we could handle a proportionally higher crawl rate, say 10 or 20 requests per second. That is, unless one of the other system administrators wants to disagree.
We've upped the rate a bit. Please let me know if it causes any problems.
(Also, is there an official contact for this sort of thing? That is: who knows / gets to make the final decision of whether wikipedia can handle more load?)