On Wed, 12 Mar 2003, Leonard Tulipan wrote:
Well Brian, off-peak hours is a bit of problem with an international website, isn't it?
No, not really. 02:00-14:00 UTC there is overall lower traffic than 14:00-02:00. "Off-peak" isn't the same as "no one at all uses the site".
However, I've heard back from the Google folks and they apparently have no such configuration option for googlebot to limit the times when particular domains are spidered:
Hi Brion,
Thanks for contacting Google.
We apologize for the load you are experiencing on your web servers. The crawling of websites by Googlebot is an automated process, therefore we are unable to make manual requests for Googlebot. What we can offer to do is to slow down the crawl rate for your site. If you would like us to do so, please advise and we will honor your request to do so. In the meantime, if you would prefer not to have Googlebot crawl your website, please refer to our FAQ at http://www.google.com/webmasters/3.html#B3 . We hope this information is helpful. Please contact us if you have any further questions.
Regards, The Google Team
-- brion vibber (brion @ pobox.com)