[Wikipedia-l] Tentative spiders and bots policy

[Michael R. Irwin]

Brion VIBBER wrote:
> 
> Since I've unilaterally banned a couple of IPs from accessing the site
> due to unfriendly spidering and extreme database-stressing, I thought it
> best that we set up some sort of actual policy for this.
> 
> Tentative beginnings:
> http://www.wikipedia.org/wiki/Wikipedia:Spiders_and_bots_policy

Looks like a good starting point.

I am technically illiterate regarding spidering
and web-bots.

It is possible to specify a maximum acceptable hit rate
quantitatively in robots.txt or should we lay out some
guidelines on the page for bot/spider adminstraters?

Regards,
Mike Irwin