On ven, 2002-05-17 at 13:04, General Wesc (LKBM) wrote:
I have to wonder though...if a spider goes to Recent Changes and then to "Last 5000 changes" (and last 90 days, and last 30 days, and last 2500 changes, and last 1000 changes, and every such combination) it seems to me the server load could get pretty high.
Does anyone _really_ ever want to look at the last 5000 changes?
As for the higher day values, they'll be useful for the less active other-language wikipedias once they're converted, though it might be good to have "intelligent" scaling on that bar. 90 days worth of changes on the English 'pedia would go well over ever the 5000 changes limit... (At the moment, 5000 only gets us back to about April 27.)
Perhaps talk pages should be spidered, but not recent changes or the history (diff/changes).
A robots.txt could easily be set up to disallow /wiki/special%3ARecentChanges (and various case variations). That only stops _nice_ spiders, of course.
History links would need to be changed to be sufficiently distinguishable, for instance using /wiki.phtml?title=Foo&action=history etc; then ban /wiki.phtml.
-- brion vibber (brion @ pobox.com)