We met briefly to weight the pros and cons of taking up a project to tag
bot traffic.
TL;DR
While we see several projects that will benefit from a more precise bot
identification we think that at this time there are workarrounds that we
can do to filter bot traffic in most areas and that we should not expend
the resources and computation effort that a through bot detection system
will require.
We think is worth spending time in quantifying our TRUE bot traffic so when
management has a question like "How much of our traffic is crawling?" we
can give an estimate by, say, researching bot traffic monthly, weekly and
daily in one given month.
At this time 15% of our pageview traffic (not requests) are detected bots,
we estimate that the real bot traffic might be quite a big higher.
More detailed Notes here:
https://wikitech.wikimedia.org/wiki/Analytics/Bots
Attendees please modify/correct notes as needed.