On Mon, 29 Sep 2003 11:07:26 -0700, Brion Vibber <brion(a)pobox.com> gave
utterance to the following:
If one of the list admins sent this through; please
don't. It's *spam*.
They crawl the web for e-mail addresses and then send these things out
as advertisements for their link-checking / availability service. Which,
of course, means they're crawling our web site and loading it down.
*Thanks* so much.
I wonder if their crawler respects robots.txt? How hard would it be to id
the crawler from server logs? Does W keep server logs?
Between two evils always pick the one you haven't tried