Accidentally i watched my logfiles (tail -f) and noticed a weird behaviour of the google spider. It spends literally days following all links in Spezial:Recentchanges.
It may be a apache configuration mistake (how?), but it also may be a mediawiki problems.
How can i disallow search engines from indexing all the recent changes? They are worthless to index anyway.
Andres Obrero
Am 16.08.2005 um 17:14 schrieb andres:
Accidentally i watched my logfiles (tail -f) and noticed a weird behaviour of the google spider. It spends literally days following all links in Spezial:Recentchanges.
It may be a apache configuration mistake (how?), but it also may be a mediawiki problems.
How can i disallow search engines from indexing all the recent changes? They are worthless to index anyway.
you can use http://en.wikipedia.org/robots.txt
ciao, tom
-- http://de.wikipedia.org/wiki/Benutzer:TomK32 http://www.tomk32.de
On Tue, 16 Aug 2005, andres wrote:
Accidentally i watched my logfiles (tail -f) and noticed a weird behaviour of the google spider. It spends literally days following all links in Spezial:Recentchanges.
there is one problem with Special:Recentchanges where you can click further and further on "next pages", even if there are no more.
How can i disallow search engines from indexing all the recent changes? They are worthless to index anyway.
check out: http://en.wikipedia.org/robots.txt for an example. You could use:
User-agent: * Disallow: /wiki/Special:Randompage Disallow: /wiki/Special%3ARandompage Disallow: /wiki/Special:Recentchanges Disallow: /wiki/Special%3ARecentchanges
christof
mediawiki-l@lists.wikimedia.org