On Tue, 11 Mar 2003, Brion Vibber wrote:
Googlebot is fairly light (several seconds to 30
seconds between requests,
and follows our robots.txt restrictions - its getting articles, not
millions of diffs or contribs pages. It's only a fraction of total pages
being served). I have written them an e-mail asking if it's possible to
restrict the spidering to off-peak hours, though.
I should point out too that I tried barring googlebot (by 403 Forbidden
reponses) for the last ten minutes or so. No improvement in response time,
so I've restored access.
-- brion vibber (brion @
pobox.com)