On Wed, 12 Mar 2003, Leonard Tulipan wrote:
Well Brian, off-peak hours is a bit of problem with an
international
website, isn't it?
No, not really. 02:00-14:00 UTC there is overall lower traffic than
14:00-02:00. "Off-peak" isn't the same as "no one at all uses the
site".
However, I've heard back from the Google folks and they apparently have no
such configuration option for googlebot to limit the times when particular
domains are spidered:
Hi Brion,
Thanks for contacting Google.
We apologize for the load you are experiencing on your web servers. The
crawling of websites by Googlebot is an automated process, therefore we
are unable to make manual requests for Googlebot. What we can offer to do
is to slow down the crawl rate for your site. If you would like us to do
so, please advise and we will honor your request to do so. In the
meantime, if you would prefer not to
have Googlebot crawl your website, please refer to our FAQ at
http://www.google.com/webmasters/3.html#B3 . We hope this information is
helpful. Please contact us if you have any further questions.
Regards,
The Google Team
-- brion vibber (brion @
pobox.com)