Andre Engels wrote:
2007/2/21, Tim Starling tstarling@wikimedia.org:
Jeremy Cushman wrote:
Right now on the English Wikipedia bots are limited to 6 edits per
minute.
Is it time to raise the permitted edit rate? Do we have the server
capacity
now to handle rapid editing?
If there is a technical problem with high rate editing then we will limit the rate by technical means. We can't rely on policy to maintain good performance. Recommended bot edit rates are there for non-technical reasons, such as to allow time for review.
Currently there is a technically imposed limit of 8 edits per minute on IPs and new accounts, to mitigate bot-driven vandalism. There is no limit on established accounts.
Does this mean I can disable the get-throttling of the bots? Or does this statement only hold for the edit limit?
Only for the edit limit. And only then because I doubt anyone will try a high enough edit rate to cause serious load on the server. If someone starts trying 100 reads per second, it's going to cause problems. And more so for 100 writes per second.
As a general rule, all bots should be single threaded. This limits the performance impact they can have on the server -- cheap requests can be done often, and expensive requests less often. If you run requests in parallel, then you're using more than your fair share of server resources.
Even single-threaded bots can cause problems if the requests are particularly long running. If you want to be polite and sleep between requests, you should throttle them in proportional to the service time -- a duty cycle, in other words. Say if a request takes 0.5s to service, you could use a duty cycle of 25%, corresponding to one request every 2 seconds. Think about your load on the server in terms of the number of threads you're tying up, on average.
The worst offenders we've discovered in recent times are the edit counters -- web based scripts that send requests to the servers often lasting minutes, with unrestricted parallelism. We were very tempted to block them all. We would have blocked them for much less if we weren't afraid having an angry mob of hundreds of Wikipedians obsessed with edit counts, descending on our door.
-- Tim Starling