On Tue, 11 Mar 2003, Lee Daniel Crocker wrote:
It appears we're being Googled this morning.
Googlebot is very
well-behaved, and I'm not sure if that's the problem or not, but
Googlebot is fairly light (several seconds to 30 seconds between requests,
and follows our robots.txt restrictions - its getting articles, not
millions of diffs or contribs pages. It's only a fraction of total pages
being served). I have written them an e-mail asking if it's possible to
restrict the spidering to off-peak hours, though.
-- brion vibber (brion @
pobox.com)