"Brion Vibber" brion@pobox.com schrieb:
robots.txt isn't based on user-agent. It works on the honor system; if the client doesn't obey robots.txt, there is no blocking caused by robots.txt.
Here's the current user-agent block list:
# we don't like these user-agents:
(...snip...)
acl stayaway browser Python-urllib
I think that this is from Rob's robot before a more specific user-agent was defined for it. As such, I don't think it has much use to keep it here.
Andre Engels