On 5/17/02 5:37 PM, "Axel Boldt" axel@uni-paderborn.de wrote:
A robots.txt could easily be set up to disallow /wiki/special%3ARecentChanges (and various case variations). That only stops _nice_ spiders, of course.
History links would need to be changed to be sufficiently distinguishable, for instance using /wiki.phtml?title=Foo&action=history etc; then ban /wiki.phtml.
I think we should do that ASAP. Let's close the whole special: namespace, &action=edit, &action=history, &diff=yes and &oldID stuff to spiders. None of this is of any value to the spiders anyway.
I think we should not do that any time soon. For one, this is a wikipedia-l level discussion. Until there is direct evidence that spiders are causing any serious problem for Wikipedia (and noone has presented any) we shouldn't even be discussing this.
Just because you can't see why it would be of value doesn't mean that it isn't.
Again, if we surmise that spiders are causing slowdowns, we should be able to find evidence for that BEFORE we block parts of the site from them. And even then we should see if the fault lies in the site's code.
Spiders simulate high traffic well, and that's something that wikipedia should be able to handle.
tc