On 10/26/07, RLS evendell@gmail.com wrote:
Anthony wrote:
On 10/26/07, Yousef Ourabi yourabi@zero-analog.com wrote:
What I don't understand is how this fits in with the "no-crawler" policy?
What "no-crawler" policy?
[[Wikipedia:Database download#Please do not use a web crawler]]
Have Google and Yahoo been informed of this policy?
BTW, that talks about articles, not images. And it contradicts robots.txt, especially "## we're disabling this experimentally 11-09-2006\n#Crawl-delay: 1"
It seems to stem from something said on the Village Pump back in 2003. I for one am going to go with robots.txt, not something someone said on some Wikipedia page.