But seriously, my tl;dr: instead of asking if an account is or isn't a bot, ask if a set of edits are or are not automated
Great responses so far: searching usernames for *bot will exclude non-bot users who were registered before the username policy change (although *Bot is a bit better), and the logging table is a great way to collect bot flags. However, Scott is right -- the bot flag (or *Bot username) doesn't signify a bot, it signifies a bureaucrat recognizing that a user account successfully went through the Bot Approval Group process. If I see an account with a bot flag, I can generally assume the edits that account makes are initiated by an automated software agent. This is especially the case in the main namespace. The inverse assumption is not nearly as easy: I can't assume that every edit made from an account *without* a bot flag was *not* an automated edit.
About unauthorized bots: yes, there are a relatively small number of Wikipedians who, on occasion, run fully-automated, continuously-operating bots without approval. Complicating this, if someone is going to take the time to build and run a bot, but isn't going to create a separate account for it, then it is likely that they are also using that account to do non-automated edits. Sometimes new bot developers will run an unauthorized bot under their own account during the initial stages of development, and only later in the process will they create a separate bot account and seek formal approval and flagging. It can get tricky when you exclude all the edits from an account for being automated based on a single suspicious set of edits.
More commonly, there are many more people who use automated batch tools like AutoWikiBrowser to support one-off tasks, like mass find-and-replace or category cleanup. Accounts powered by AWB are technically not bots, only because a human has to sit there and click "save" for every batch edit that is made. Some people will create a separate bot account for AWB work and get it approved and flagged, but many more will not bother. Then there are people using semi-automated, human-in-the-loop tools like Huggle to do vandal fighting. I find that the really hard question is whether you include or exclude these different kinds of 'cyborgs', because it really makes you think hard about what exactly you're measuring. Is someone who does a mass find-and-replace on all articles in a category a co-author of each article they edit? Is a vandal fighter patrolling the recent changes feed with Huggle a co-author of all the articles they edit when they revert vandalism and then move on to the next diff? What about somebody using rollback in the web browser? If so, what is it that makes these entities authors and ClueBot NG not an author?
When you think about it, user accounts are actually pretty remarkable in that they allow such a diverse set of uses and agents to be attributed to a single entity. So when it comes to identifying automation, I personally think it is better to shift the unit of analysis from the user account to the individual edit. A bot flag lets you assume all edits from an account are automated, but you can use a range of approaches to identifying sets of automated edits from non-flagged accounts. Then I have a set of regex SQL queries in the Query Library [1] which parses edit summaries for the traces that AWB, Huggle, Twinkle, rollback, etc. automatically leave by default. You can also use the edit session approach like Scott has suggested -- Aaron and I found a few unauthorized bots in our edit session study [2], and we were even using a more aggressive break, with no more than a 60 minute gap between edits. To catch short bursts of bulk edits, you could look at large numbers of edits made in a short period of time -- I'd say more than 7 main namespace edits a minute for 10 minutes would be a hard rate for even a very aggressive vandal fighter to maintain with Huggle.
I'll conclude by saying that different kinds of automated editing techniques are different ways of participating in and contributing to Wikipedia. To systematically exclude automated edits is to remove a very important, meaningful, and heterogeneous kind of activity from view. These activities constitute a core part of what Wikipedia is, particularly those forms of automation which the community has explicitly authorized and recognized. Now, we researchers inevitably have to selectively reveal and occlude -- a co-authorship network based on main namespace edits also excludes talk page discussions and conflict resolution, and this also constitutes a core part of what Wikipedia is. It isn't wrong per se to exclude automated edits, and it is certainly much worse to not recognize that they exist at all. However, I always appreciate seeing how the analysis would be different if bots were not excluded. The fact that there are these weird users which absolutely dominate a co-authorship network graph if you don't filter them out is pretty amazing, at least to me.
Best,
Stuart