"Brion Vibber" brion@pobox.com schrieb:

robots.txt isn't based on user-agent. It works on the honor system; if the client doesn't obey robots.txt, there is no blocking caused by robots.txt.

Here's the current user-agent block list:

# we don't like these user-agents:

(...snip...)

acl stayaway browser Python-urllib

I think that this is from Rob's robot before a more specific user-agent was defined for it. As such, I don't think it has much use to keep it here.

Andre Engels