On Fri, Feb 28, 2014 at 6:29 PM, Brad Jorsch (Anomie) <bjorsch@wikimedia.org
wrote:
If you display 8 images and the user has to pick one, then even by random guessing the attacker has a 12.5% chance of passing the captcha. That's not good at all. Finding "all matching" is slightly better since it reduces the guessability (1/256 for 8 images), but still not very good. A traditional captcha using only A-Z is 1/308915776. To do as well with image picking, you'd need to ask the user to choose the matches from a set of about 28. Adding in numbers 2-9 is 1/1544804416, needing a set of about 31 images.
The set of possible images also needs to be very large and the categorization private.
https://www.mediawiki.org/wiki/Talk:Requests_for_comment/CAPTCHA#Issue:_imag... into much more detail on this issue.
A recent example that springs to mind with image-based CAPTCHAs (instead of text) is Snapchat's "Find the Ghost", which is very fun for users and apparently was broken very quickly.[1] A lot of times I hear people also suggest we try a honeypot on login/signup instead of text-based CAPTCHAs, and like the Snapchat example, one of the weaknesses here is just not accounting for that fact that people will target popular sites/apps directly. They'll inspect the DOM to find honeypots, they'll notice you use the same logo shape and use computer vision to find that shape, etc.
However, it is not overstating it to say that the text-based CAPTCHA we use now is the single most frustrating part of creating an account or logging in (if you misremember your password, which users do all the time). To quote one of our usability tests during the last login/signup redesign: "This is ridiculous. I can't even see this.".[2]
One simpler thing we might try and do right now is regenerate our current pool of CAPTCHAs to make them a bit less hard to read. We've done this kind of tweaking before without too much trouble I think?[3]
1. techcrunch.com/2014/01/21/snaptcha/ 2. https://www.mediawiki.org/wiki/Account_creation_user_experience/User_testing 3. See bug 43546 which Aaron Schulz kindly took care of. He may be able to elaborate more.