On 29/04/05, Jeremie Bouillon jeremie@shadowfrance.net wrote:
About the robot.txt, can we find the one used by large wiki (like Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from where there aren't supposed to be ?
robots.txt is just a plain text file in the root of the webserver; you can look at anybody's in a browser by typing the URL in e.g. http://en.wikipedia.org/robots.txt http://www.google.com/robots.txt or even http://www.robotstxt.org/robots.txt :)