Thanks so much!
Just to give a little more detail on user selection in usertesting.com
, it seems that aside from very general demographic preferences, basically age range and gender, it is possible to specify highest rated testers and favorite testers.
By not selecting any preferences, it seems we can get a diverse sampling of skill levels, and perhaps even more importantly, the testers are not sampled from our "normal" audience. The net effect is more like that of a focus group, but with the key difference that users are in their natural setting using their own devices (along the lines of country-specific user studies the Foundation has been performing in recent years).
Thanks for being helpful (here and in mobile development).
Phil Inje Chang
Product Manager, Mobile
415-882-7982 x 6810