$ GET
http://lists.wikimedia.org/robots.txt |sed 's/^/>/'
# robots.txt for
lists.wikimedia.org
#
# Disabled crawling for several lists 2005-11-26 to
# discourage people from complaining about items they
# post on public mailing lists being the first Google
# search result about them.
Tell them to use
http://en.wikipedia.org/wiki/X-No-Archive next time.
Then there would be no need to make all the other people's posts
unsearchable too, and it will even work on all the other sites the lists
are archived, e.g.,
gmane.org.
# Disabled for all lists 2006-11-03, now that an
internal
# search has been set up using htdig.
I can't find the URL for that.
# Note that list archives remain public.
Public but not searchable?!
User-agent: *
Disallow: /pipermail/