Dan Jacobson wrote:
Simetrical says
S> Why? noindex,follow for Allpages strikes me as sensible, even if not as useful
S> as a site map.
It's a deal. However,
$ GET
http://taizhongbus.jidanni.org/index.php?title=%E7%89%B9%E6%AE%8A:Allpages|…
robot
<meta name="robots" content="noindex,nofollow" />
as of 1.8.2.
The inherent problem with Special pages is that they're dynamically
generated. Thus, the crawler can needlessly crawl (and index)
approximately the same content repeatedly. For example, it could index
both
http://en.wikipedia.org/w/index.php?title=Special:Allpages&from=The_Los…
and
http://en.wikipedia.org/w/index.php?title=Special:Allpages&from=The%20L…
, even though they're essentially identical. Really, we don't want
these types of pages to appear in the search engine at all. If we want
to implement Sitemap, we should do it right.
Matthew Flaschen