No, it is not, and in my opinion it would also not be desirable to do
so. Among the forbids in the robots.txt is the forbid to get edit
pages; without that it would be impossible to run the bot. Under
normal circumstances the bot will keep to the specification in
robots.txt that there has to be at least 1 second between two
requests.
Andre Engels
On 7/11/05, Scot Wilcoxon <scot(a)wilcoxon.org> wrote:
Do wikipedia.py and urllib2 obey robots.txt? I
didn't find anything in
the former.