Dan Jacobson wrote:
Simetrical says S> Why? noindex,follow for Allpages strikes me as sensible, even if not as useful S> as a site map.
It's a deal. However, $ GET http://taizhongbus.jidanni.org/index.php?title=%E7%89%B9%E6%AE%8A:Allpages%7... robot
<meta name="robots" content="noindex,nofollow" /> as of 1.8.2.
The inherent problem with Special pages is that they're dynamically generated. Thus, the crawler can needlessly crawl (and index) approximately the same content repeatedly. For example, it could index both http://en.wikipedia.org/w/index.php?title=Special:Allpages&from=The_Lost... and http://en.wikipedia.org/w/index.php?title=Special:Allpages&from=The%20Lo... , even though they're essentially identical. Really, we don't want these types of pages to appear in the search engine at all. If we want to implement Sitemap, we should do it right.
Matthew Flaschen