Yeh I had thought I was being sensitive with my choices and trying to
carefully craft my questions. For instance I did not once mention
background but did gently ask where he/she was based. Obviously I
sadly failed. :(
I understand there are problems with an A/B test but I disagree that
usertesting would give a better idea about this problem. If the A/B
test is limited to anonymous users on all pages, then I would expect
us to still be able to deduce whether minor changes to the UI
encourage clicking (in an audience if 30% of that has never clicked
the icon we would still see differences in click through rate in an
A/B test as 15% of those would be captured in the A/B test).
I think usertesting would actually be a worse approach as you are then
basing your decision on the views of a small amount of people. I also
think the environment created by
encourages clicking
around and doing things you wouldn't normally do (due to the fact you
are aware you are trying to give value to whoever is viewing the
result).
On Fri, Nov 15, 2013 at 1:08 PM, Steven Walling <swalling(a)wikimedia.org> wrote:
On Fri, Nov 15, 2013 at 11:33 AM, Jon Robson <jrobson(a)wikimedia.org> wrote:
May and I tried to probe a bit more to understand whether it's a
cultural thing (e.g. maybe the hamburger icon is only recognisable in
North America/Western Europe). Alas, he/she then got very defensive
and angry and for some reason didn't want to talk about it.
Yeah it's generally not a good idea to ask questions than can (even vaguely)
imply a problem is the user's fault, and doubly so if you're suggesting it's
because they come from [insert cultural background here].
Ċ½eljko: we call it the hamburger as a joke. It's supposed to represent a
list of menu items.
He did however mention she/he wanted a help page but when I asked
where that would be discovered he went into radio silence.
I know the design team have been looking to move away from the
hamburger icon and after talking with May we reckon it might be useful
to run a few A/B tests around it to see if it becomes 'more clickable'
if
* a divider is present
* if we try effects on the icon to make it look more tappable
* If it changes to an icon other than the hamburger
The three variables you propose to A/B test are very difficult to create a
valid controlled study around. You have to include only users who have never
seen the current mobile site, because otherwise you're creating a huge
confound by including users who remember the placement and purpose of the
current icon. Even if you change the icon entirely for everyone, users
remember placement and use it to orient themselves within an interface just
as much as icons or other details. I would suggest solving this problem with
a more qualitative approach, ala
usertesting.com remote tests.
--
Steven Walling,
Product Manager
https://wikimediafoundation.org/