On 7/5/07, jidanni@jidanni.org jidanni@jidanni.org wrote:
No wonder my wiki is so popular when measured according to bandwidth needs. It's all brand name searchengines sniffing up those non-existent page links. I'm not sure what ought to be done.
==> logs/radioscanningtw.jidanni.org/http.2880196/access.log <== 74.6.20.76 - - [05/Jul/2007:19:35:33 -0700] "GET /index.php?title=Talk:%E5%8F%B0%E4%B8%AD%E7%B8%A3%E8%AD%A6%E5%AF%9F%E5%B1%80%E6%9D%B1%E5%8B%A2%E5%88%86%E5%B1%80&action=edit HTTP/1.0" 200 4896 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
robots.txt:
Disallow: /index.php
Assuming you're using URL rewriting, this will have the robots request only actual wiki pages, not edit pages/history pages/etc. You may also wish to blacklist Special:Random.