[Wikipedia-l] Maintenance page and web spiders

[Neil Harris]

Khendon wrote:

>On Thu, Oct 10, 2002 at 10:28:00AM +0100, Neil Harris wrote:
>  
>
>>The new maintenance page is great! But the links on it should be 
>>disabled for non logged-in users so that web spiders don't exercise the 
>>high-overhead functions on the maintenance page, bogging down the server.
>>    
>>
>
>This might be better handled with robots.txt...
>
>  
>
I'd do both. Robots.txt only keeps out well-behaved robots, but it's the 
badly-behaved robots that cause most real trouble. And if they are badly 
behaved enough to hammer links, they are probably also badly behaved 
enough to ignore robots.txt.

Neil