I've noticed that Wikipedia now adds rel="nofollow" to user contributed URLs, so that [http://myurl.com/ website] becomes <a href="http://myurl.com/" rel="nofollow" >website</a>.
The rel="nofollow" attribute was (unofficially?) announced by Google in January this year, as a means to fight link spam in blogs, wikis and guest books.
But shouldn't rel="nofollow" also be used for the edit, history and diff links? These pages avoid Google indexing by having the <meta name="robots" content="noindex,nofollow" /> tag in their <head> section, but Google only sees that after having visited the page. The visits by Google to these pages are not productive, but still add load on Wikipedia's servers, and this could be avoided by adding rel=nofollow on the edit, history and diff links.
-- Lars Aronsson (lars@aronsson.se) Aronsson Datateknik - http://aronsson.se
Lars Aronsson wrote in news.gmane.org.wikimedia.mediawiki:
But shouldn't rel="nofollow" also be used for the edit, history and diff links? These pages avoid Google indexing by having the
<meta name="robots" content="noindex,nofollow" /> tag in their <head> section, but Google only sees that after having visited the page. The visits by Google to these pages are not productive, but still add load on Wikipedia's servers, and this could be avoided by adding rel=nofollow on the edit, history and diff links.
i didn't look at nofollow too closely, but my impression was that (despite the name), Google would still follow the link, but not assign it any pagerank based on that particular link.
in any case, generated pages should be denied to Google via robots.txt:
http://en.wikipedia.org/robots.txt
kate.
(cue discussion about whether nofollow is the devil and will destroy the intarweb.)
Le 23:08 29/04/2005, Kate Turner écrivait :
i didn't look at nofollow too closely, but my impression was that (despite the name), Google would still follow the link, but not assign it any pagerank based on that particular link.
That's my feeling too. After the first publication, I email them (twice) about this, asking if GoogleBot would not follow or simply not index, and if they might consider making it no follow if that wasn't the case (or add another attribute). Never got a answer.
About the robot.txt, can we find the one used by large wiki (like Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from where there aren't supposed to be ?
Jeremie Bouillon wrote:
About the robot.txt, can we find the one used by large wiki (like Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from where there aren't supposed to be ?
It's in the customary location, of course: http://en.wikipedia.org/robots.txt
robots.txt files are fun... If you look at http://www.whitehouse.gov/robots.txt you'll find the White House is hiding WMDs!!!! ("Disallow: /wmd/text")
Other people just don't like having their documents archived for later reference: http://www.sco.com/robots.txt
Not that some people should complain... ;) http://www.groklaw.net/robots.txt
-- brion vibber (brion @ pobox.com)
Le 00:25 30/04/2005, Brion Vibber écrivait :
Jeremie Bouillon wrote:
About the robot.txt, can we find the one used by large wiki (like Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from where there aren't supposed to be ?
It's in the customary location, of course: http://en.wikipedia.org/robots.txt
I should buy a new brain. Anyone saw a cheap one on Ebay recently ?
Thanks :)
I should buy a new brain. Anyone saw a cheap one on Ebay recently ?
I'm selling mine but it's marked as faulty - for spare parts only.
Jeremie Bouillon wrote:
Le 00:25 30/04/2005, Brion Vibber écrivait :
Jeremie Bouillon wrote:
About the robot.txt, can we find the one used by large wiki (like Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from where there aren't supposed to be ?
It's in the customary location, of course: http://en.wikipedia.org/robots.txt
I should buy a new brain. Anyone saw a cheap one on Ebay recently ?
http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&category=11737&item=59...
Go quick, the bidding ends in an hour or so!
-- brion vibber (brion @ pobox.com)
Of course, if you converted your existing brain to a wiki, you could plug it into the net, and everyone can make changes they think will be an improvement... and this would save the cost of a new brain!
Jeremie Bouillon wrote: I should buy a new brain. Anyone saw a cheap one on Ebay recently ?
Thanks :)
On 29/04/05, Jeremie Bouillon jeremie@shadowfrance.net wrote:
About the robot.txt, can we find the one used by large wiki (like Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from where there aren't supposed to be ?
robots.txt is just a plain text file in the root of the webserver; you can look at anybody's in a browser by typing the URL in e.g. http://en.wikipedia.org/robots.txt http://www.google.com/robots.txt or even http://www.robotstxt.org/robots.txt :)
Lars Aronsson wrote:
But shouldn't rel="nofollow" also be used for the edit, history and diff links? These pages avoid Google indexing by having the
<meta name="robots" content="noindex,nofollow" /> tag in their <head> section, but Google only sees that after having visited the page. The visits by Google to these pages are not productive, but still add load on Wikipedia's servers, and this could be avoided by adding rel=nofollow on the edit, history and diff links.
They don't add load on Wikipedia's servers because they're locked off by our robots.txt.
It's also not clear whether or not rel="nofollow" actually tells Google not to spider the page, or simply not to count the link for pagerank. If you can find a clear spec for this, let me know.
-- brion vibber (brion @ pobox.com)
On 4/29/05, Lars Aronsson lars@aronsson.se wrote:
But shouldn't rel="nofollow" also be used for the edit, history and diff links? These pages avoid Google indexing by having the
<meta name="robots" content="noindex,nofollow" /> tag in their <head> section, but Google only sees that after having visited the page. The visits by Google to these pages are not productive, but still add load on Wikipedia's servers, and this could be avoided by adding rel=nofollow on the edit, history and diff links.
Edit, history, and diff pages are served from /w/ urls, which robots.txt disallows for all user-agents that obey the robots protocol. So google doesn't visit these pages at all.
mediawiki-l@lists.wikimedia.org