On Mon, 29 Sep 2003, Vicki Rosenzweig wrote:
At 09:22 AM 9/30/03 +1200, Richard Grevers wrote:
I wonder if their crawler respects robots.txt?
I would assume not--robots.txt is a protocol for honest people, not
spammers and liars.
They claim it does. They did hit robots.txt, and their accesses today seem
to be within the general settings (no access to /w/wiki.phtml?whatever).
I've changed it to now disallow any access by them on their next visit,
we'll see what happens.
-- brion vibber (brion @
pobox.com)