On Oct 26, 2004, at 4:04 PM, Michael wrote:
It might be useful to include a robots.txt file that'll tell search spiders not to bother with any of the active pages such as 'Edit'.
This will be dependent on your server configuration. Note that robots.txt works on URL prefixes, so you need a reliable way of distinguishing plain view hits from other URLs.
(The meta tags already tell search engines not to index edit pages and other special pages, and not to continue spidering from them, but won't prevent the initial hit to load that page.)
-- brion vibber (brion @ pobox.com)