[Labs-l] Google bot

[Magnus Manske]

Hi,

I saw a high load (dozens of queries [1]) hitting one of my tools
(catscan2). The queries looked like they came from a template on French
Wikipedia (category name different, other parameters the same). Access log
shows (among other things) Google bot. When I added that to my bot
exclusion list in .lighttpd.conf and restated, the number of queries went,
and so far stayed, down.

I vaguely remember that indexing bots (like the Google one) were filtered
out by Labs already? If it does, was that query drop random then?

Cheers,
Magnus

[1] https://tools.wmflabs.org//catscan2/server-status
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20141019/3128f560/attachment.html>