Jeremie Bouillon wrote:
About the robot.txt, can we find the one used by large wiki (like Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from where there aren't supposed to be ?
It's in the customary location, of course: http://en.wikipedia.org/robots.txt
robots.txt files are fun... If you look at http://www.whitehouse.gov/robots.txt you'll find the White House is hiding WMDs!!!! ("Disallow: /wmd/text")
Other people just don't like having their documents archived for later reference: http://www.sco.com/robots.txt
Not that some people should complain... ;) http://www.groklaw.net/robots.txt
-- brion vibber (brion @ pobox.com)