On Friday 21 June 2002 09:02 am, tarquin wrote:

might Google also be following "pages that link here", "random" and "history"? Wikipedia freezes almost every day around 3pm UK time, which is odd as it's during the pediod with the fewest edits. Could that be when google comes a-crawling? we had problems on UnrealWiki with downloading robots calling too many diff links & oveerloading the server.

tarquin

Is this fixed in the new code? I thought we decided not to allow Google to index edit links (along with older versions of articles in History, log pages and special pages other than maybe RecentChanges)? I don't see a problem with allowing Google to follow links through "pages that link here" though -- so long as this doesn't cause undue hardship on the server (if anything, this should decrease the amount of time it take Googlebot to do its thing).

maveric149