Yeh I had thought I was being sensitive with my choices and trying to carefully craft my questions. For instance I did not once mention background but did gently ask where he/she was based. Obviously I sadly failed. :(
I understand there are problems with an A/B test but I disagree that usertesting would give a better idea about this problem. If the A/B test is limited to anonymous users on all pages, then I would expect us to still be able to deduce whether minor changes to the UI encourage clicking (in an audience if 30% of that has never clicked the icon we would still see differences in click through rate in an A/B test as 15% of those would be captured in the A/B test).
I think usertesting would actually be a worse approach as you are then basing your decision on the views of a small amount of people. I also think the environment created by usertesting.com encourages clicking around and doing things you wouldn't normally do (due to the fact you are aware you are trying to give value to whoever is viewing the result).
On Fri, Nov 15, 2013 at 1:08 PM, Steven Walling swalling@wikimedia.org wrote:
On Fri, Nov 15, 2013 at 11:33 AM, Jon Robson jrobson@wikimedia.org wrote:
May and I tried to probe a bit more to understand whether it's a cultural thing (e.g. maybe the hamburger icon is only recognisable in North America/Western Europe). Alas, he/she then got very defensive and angry and for some reason didn't want to talk about it.
Yeah it's generally not a good idea to ask questions than can (even vaguely) imply a problem is the user's fault, and doubly so if you're suggesting it's because they come from [insert cultural background here].
Ċ½eljko: we call it the hamburger as a joke. It's supposed to represent a list of menu items.
He did however mention she/he wanted a help page but when I asked where that would be discovered he went into radio silence.
I know the design team have been looking to move away from the hamburger icon and after talking with May we reckon it might be useful to run a few A/B tests around it to see if it becomes 'more clickable' if
- a divider is present
- if we try effects on the icon to make it look more tappable
- If it changes to an icon other than the hamburger
The three variables you propose to A/B test are very difficult to create a valid controlled study around. You have to include only users who have never seen the current mobile site, because otherwise you're creating a huge confound by including users who remember the placement and purpose of the current icon. Even if you change the icon entirely for everyone, users remember placement and use it to orient themselves within an interface just as much as icons or other details. I would suggest solving this problem with a more qualitative approach, ala usertesting.com remote tests.
-- Steven Walling, Product Manager https://wikimediafoundation.org/