On 17/08/07, jidanni@jidanni.org jidanni@jidanni.org wrote:
Anyway of keeping these from wasting bandwidth, GET /index.php?title=Special:Search&ns0=1&redirs=0&searchx=1&search=487.3000 "Baiduspider GET /index.php?title=Category:130.6000&action=edit Googlebot/2.1 _without_ worrying about making pretty URLs (so one can use "index.php" in robots.txt)?
Why can't there be a nofollow added to such links next version?
No doubt we can add rel="nofollow" to broken links, but be aware that robots are not required to adhere to this; the actual meaning is not "don't follow this link", but rather, "don't afford the page this links to any significance". [see http://microformats.org/wiki/rel-nofollow]
At present, the only way to instruct a robots exclusion standard compliant (robots.txt-compliant) robot not to follow links on a page is via an appropriate line in said robots.txt file, or using the <meta name="robots"> tag, setting the "content" attribute to contain "nofollow". And of course, it's not possible to *enforce* that this is followed without resorting to crude access control.
Rob Church