On 14/05/06, Neil Harris neil@tonal.clara.co.uk wrote:
Version 4: Google has added special-case code to their crawler engine that scan, say, Recent Changes, and spiders new material in near-real-time.
This would require them to start ignoring meta tags containing instructions to robots, since recent changes (and other special pages) have "noindex,nofollow" set.
Google are teh evil!
Rob Church