Brion VIBBER wrote:
Since I've unilaterally banned a couple of IPs from accessing the site due to unfriendly spidering and extreme database-stressing, I thought it best that we set up some sort of actual policy for this.
Tentative beginnings: http://www.wikipedia.org/wiki/Wikipedia:Spiders_and_bots_policy
Looks like a good starting point.
I am technically illiterate regarding spidering and web-bots.
It is possible to specify a maximum acceptable hit rate quantitatively in robots.txt or should we lay out some guidelines on the page for bot/spider adminstraters?
Regards, Mike Irwin