I noticed today that "TweetmemeBot", a web crawler, was making lots of
requests to my tools and ignoring the robots.txt file. The thing
scrapes at a very high rate, sometimes over 30 requests per minute.
Other tool maintainers (especially geohack) may want to investigate
whether this is also impacting their tools, and explicitly block the
user agent if it is.
- Carl
Show replies by date