On 5/17/02 5:37 PM, "Axel Boldt" <axel(a)uni-paderborn.de> wrote:
A robots.txt
could easily be set up to disallow
/wiki/special%3ARecentChanges (and various case variations). That only
stops _nice_ spiders, of course.
History links would need to be changed to be
sufficiently
distinguishable, for instance using
/wiki.phtml?title=Foo&action=history
etc; then ban /wiki.phtml.
I think we should do that ASAP. Let's close the whole special:
namespace, &action=edit, &action=history, &diff=yes and &oldID stuff
to spiders. None of this is of any value to the spiders anyway.
I think we should not do that any time soon. For one, this is a wikipedia-l
level discussion. Until there is direct evidence that spiders are causing
any serious problem for Wikipedia (and noone has presented any) we shouldn't
even be discussing this.
Just because you can't see why it would be of value doesn't mean that it
isn't.
Again, if we surmise that spiders are causing slowdowns, we should be able
to find evidence for that BEFORE we block parts of the site from them. And
even then we should see if the fault lies in the site's code.
Spiders simulate high traffic well, and that's something that wikipedia
should be able to handle.
tc