On 14/05/06, Neil Harris <neil(a)tonal.clara.co.uk> wrote:
Version 4: Google has added special-case code to their
crawler engine
that scan, say, Recent Changes, and spiders new material in near-real-time.
This would require them to start ignoring meta tags containing
instructions to robots, since recent changes (and other special pages)
have "noindex,nofollow" set.
Google are teh evil!
Rob Church