On Tue, Apr 15, 2014 at 10:12 AM, Steven Walling steven.walling@gmail.comwrote:
Are you volunteering to build such a survey tool? ;-)
Will see if I find the time. "Survey" probably gives the wrong idea here, it is really just an overlay with two buttons, more of an interactive A/B test. Could be probably cobbled together from GuidedTours and EventLogging.
When it comes to using a survey to catch problems early and gauging preferences, a survey still very much suffers from the self-selection bias that all opt-in options have. It's just the name of the game. When you move something from opt-in to opt-out you reach a wider audience and encounter new complaints/questions/bugs.
You can survey the opt-out audience before actually enabling any changes; that is a good way of catching those bugs without actually causing them. Of course, that point is moot now, and the refresh seemed like a simple change without the benefit of hindsight.
Still, it might be useful to run such a survey (or surveys) even now:
- Which fonts users would prefer is mostly based on educating guesses now. Complaints and bugs are much more heavily self-selected than a survey (especially a super-short one-click survey), so even though the results would still be slanted towards more active users, you would get a better picture of severity. - There is a lot of uncertainty about how widespread certain bugs are (e.g. ClearText issues); showing an affected text and asking "Does this look good to you?" is an easy way to get data about that.