Do we forbid certain spiders access to the site based on User-Agent? A user in a German forum reported recently that he couldn't access Wikipedia at all, always receiving a "Forbidden" message. It turned out that his webwasher proxy was to blame (an ad banner block). The proxy sends the User-Agent
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt) WebWasher 3.0"
Webwasher cannot be used to spider and download sites.
Axel