On Tue, Apr 15, 2014 at 10:12 AM, Steven Walling
<steven.walling(a)gmail.com>wrote;wrote:
Are you volunteering to build such a survey tool? ;-)
Will see if I find the time. "Survey" probably gives the wrong idea here,
it is really just an overlay with two buttons, more of an interactive A/B
test. Could be probably cobbled together from GuidedTours and EventLogging.
When it comes to using a survey to catch problems
early and gauging
preferences, a survey still very much suffers from the self-selection bias
that all opt-in options have. It's just the name of the game. When you move
something from opt-in to opt-out you reach a wider audience and encounter
new complaints/questions/bugs.
You can survey the opt-out audience before actually enabling any changes;
that is a good way of catching those bugs without actually causing them.
Of course, that point is moot now, and the refresh seemed like a simple
change without the benefit of hindsight.
Still, it might be useful to run such a survey (or surveys) even now:
- Which fonts users would prefer is mostly based on educating guesses
now. Complaints and bugs are much more heavily self-selected than a survey
(especially a super-short one-click survey), so even though the results
would still be slanted towards more active users, you would get a better
picture of severity.
- There is a lot of uncertainty about how widespread certain bugs are
(e.g. ClearText issues); showing an affected text and asking "Does this
look good to you?" is an easy way to get data about that.