On 17/08/07, jidanni(a)jidanni.org <jidanni(a)jidanni.org> wrote:
Anyway of keeping these from wasting bandwidth,
GET
/index.php?title=Special:Search&ns0=1&redirs=0&searchx=1&search=487.3000
"Baiduspider
GET /index.php?title=Category:130.6000&action=edit Googlebot/2.1
_without_ worrying about making pretty URLs (so one can use "index.php" in
robots.txt)?
Why can't there be a nofollow added to such links next version?
No doubt we can add rel="nofollow" to broken links, but be aware that
robots are not required to adhere to this; the actual meaning is not
"don't follow this link", but rather, "don't afford the page this
links to any significance". [see
http://microformats.org/wiki/rel-nofollow]
At present, the only way to instruct a robots exclusion standard
compliant (robots.txt-compliant) robot not to follow links on a page
is via an appropriate line in said robots.txt file, or using the <meta
name="robots"> tag, setting the "content" attribute to contain
"nofollow". And of course, it's not possible to *enforce* that this is
followed without resorting to crude access control.
Rob Church