"Brion Vibber" <brion(a)pobox.com> schrieb:
robots.txt isn't based on user-agent. It works on
the honor system; if
the client doesn't obey robots.txt, there is no blocking caused by
robots.txt.
Here's the current user-agent block list:
# we don't like these user-agents:
(...snip...)
acl stayaway browser Python-urllib
I think that this is from Rob's robot before a more specific user-agent
was defined for it. As such, I don't think it has much use to keep it
here.
Andre Engels