On Friday 21 June 2002 09:02 am, tarquin wrote:
might Google also be following "pages that link
here", "random" and
"history"?
Wikipedia freezes almost every day around 3pm UK time, which is odd as
it's during the pediod with the fewest edits. Could that be when google
comes a-crawling?
we had problems on UnrealWiki with downloading robots calling too many
diff links & oveerloading the server.
tarquin
Is this fixed in the new code? I thought we decided not to allow Google to
index edit links (along with older versions of articles in History, log pages
and special pages other than maybe RecentChanges)? I don't see a problem with
allowing Google to follow links through "pages that link here" though -- so
long as this doesn't cause undue hardship on the server (if anything, this
should decrease the amount of time it take Googlebot to do its thing).
maveric149