On Mon, 29 Sep 2003, Vicki Rosenzweig wrote:
At 09:22 AM 9/30/03 +1200, Richard Grevers wrote:
I wonder if their crawler respects robots.txt?
I would assume not--robots.txt is a protocol for honest people, not spammers and liars.
They claim it does. They did hit robots.txt, and their accesses today seem to be within the general settings (no access to /w/wiki.phtml?whatever). I've changed it to now disallow any access by them on their next visit, we'll see what happens.
-- brion vibber (brion @ pobox.com)