Simetrical says S> Why? noindex,follow for Allpages strikes me as sensible, even if not as useful S> as a site map.
It's a deal. However, $ GET http://taizhongbus.jidanni.org/index.php?title=%E7%89%B9%E6%AE%8A:Allpages%7... robot <meta name="robots" content="noindex,nofollow" /> as of 1.8.2.
Dan Jacobson wrote:
Simetrical says S> Why? noindex,follow for Allpages strikes me as sensible, even if not as useful S> as a site map.
It's a deal. However, $ GET http://taizhongbus.jidanni.org/index.php?title=%E7%89%B9%E6%AE%8A:Allpages%7... robot
<meta name="robots" content="noindex,nofollow" /> as of 1.8.2.
The inherent problem with Special pages is that they're dynamically generated. Thus, the crawler can needlessly crawl (and index) approximately the same content repeatedly. For example, it could index both http://en.wikipedia.org/w/index.php?title=Special:Allpages&from=The_Lost... and http://en.wikipedia.org/w/index.php?title=Special:Allpages&from=The%20Lo... , even though they're essentially identical. Really, we don't want these types of pages to appear in the search engine at all. If we want to implement Sitemap, we should do it right.
Matthew Flaschen
For my flat structured site with 1500 purposely empty categories, I have figured out a way to get search engines to index it.
I transclude parts of Special:Allpages to http://radioscanningtw.jidanni.org/index.php?title=%E6%A8%A3%E6%9D%BF%E8%A8%... which I link from my user page which is linked from the main page.
wikitech-l@lists.wikimedia.org