Very nice of them to let us know!

----- Forwarded message from Warren Brown wbrown@inktomi.com -----

From: "Warren Brown" wbrown@inktomi.com Date: Mon, 18 Aug 2003 17:25:08 -0700 To: root@wikipedia.org Subject: Inktomi web crawler

The wikipedia.org server is blocking Inktomi's "Slurp" web crawler by returning 403 errors for all access attempts. Presumably, this block was set up because we were crawling the site too aggressively at some time in the past. We would like to include wikipedia.org content in our search database, and would be happy to work with you to match whatever crawling limits you need to set.

Slurp observes /robots.txt rules for user-agent "Slurp". The crawler access rate is normally limited to 4 pages per minute from a web server; we can set that rate lower if you require. The Slurp access rate can also be controlled by a "crawldelay" instruction in /robots.txt.

Inktomi search service is used MSN Search and a number of other web portal and business sites worldwide. We are now a subsidiary company of Yahoo!

Regards, Warren Brown Partner Service and Support Inktomi, a Yahoo! Company

----- End forwarded message -----