We had an interesting discussion in #wikimedia-mobile today. A user came into the channel and made some comment about "nothing loads on mobile, afaik it doesnt even allow logging in"
At first I thought he/she was talking about the app but as we talked some more it became clear he/she hadn't realised the hamburger icon was clickable and had assumed it was part of the search feature.
May and I tried to probe a bit more to understand whether it's a cultural thing (e.g. maybe the hamburger icon is only recognisable in North America/Western Europe). Alas, he/she then got very defensive and angry and for some reason didn't want to talk about it.
He did however mention she/he wanted a help page but when I asked where that would be discovered he went into radio silence.
I know the design team have been looking to move away from the hamburger icon and after talking with May we reckon it might be useful to run a few A/B tests around it to see if it becomes 'more clickable' if * a divider is present * if we try effects on the icon to make it look more tappable * If it changes to an icon other than the hamburger
It also suggests we may want to think about how we surface a help feature to these users... putting it in the hamburger menu itself might not be a good idea..
On Fri, Nov 15, 2013 at 11:33 AM, Jon Robson jrobson@wikimedia.org wrote:
May and I tried to probe a bit more to understand whether it's a cultural thing (e.g. maybe the hamburger icon is only recognisable in North America/Western Europe). Alas, he/she then got very defensive and angry and for some reason didn't want to talk about it.
Yeah it's generally not a good idea to ask questions than can (even vaguely) imply a problem is the user's fault, and doubly so if you're suggesting it's because they come from [insert cultural background here].
Željko: we call it the hamburger as a joke. It's supposed to represent a list of menu items.
He did however mention she/he wanted a help page but when I asked where that would be discovered he went into radio silence.
I know the design team have been looking to move away from the hamburger icon and after talking with May we reckon it might be useful to run a few A/B tests around it to see if it becomes 'more clickable' if
- a divider is present
- if we try effects on the icon to make it look more tappable
- If it changes to an icon other than the hamburger
The three variables you propose to A/B test are very difficult to create a valid controlled study around. You have to include only users who have never seen the current mobile site, because otherwise you're creating a huge confound by including users who remember the placement and purpose of the current icon. Even if you change the icon entirely for everyone, users remember placement and use it to orient themselves within an interface just as much as icons or other details. I would suggest solving this problem with a more qualitative approach, ala usertesting.com remote tests.
Yeh I had thought I was being sensitive with my choices and trying to carefully craft my questions. For instance I did not once mention background but did gently ask where he/she was based. Obviously I sadly failed. :(
I understand there are problems with an A/B test but I disagree that usertesting would give a better idea about this problem. If the A/B test is limited to anonymous users on all pages, then I would expect us to still be able to deduce whether minor changes to the UI encourage clicking (in an audience if 30% of that has never clicked the icon we would still see differences in click through rate in an A/B test as 15% of those would be captured in the A/B test).
I think usertesting would actually be a worse approach as you are then basing your decision on the views of a small amount of people. I also think the environment created by usertesting.com encourages clicking around and doing things you wouldn't normally do (due to the fact you are aware you are trying to give value to whoever is viewing the result).
On Fri, Nov 15, 2013 at 1:08 PM, Steven Walling swalling@wikimedia.org wrote:
On Fri, Nov 15, 2013 at 11:33 AM, Jon Robson jrobson@wikimedia.org wrote:
May and I tried to probe a bit more to understand whether it's a cultural thing (e.g. maybe the hamburger icon is only recognisable in North America/Western Europe). Alas, he/she then got very defensive and angry and for some reason didn't want to talk about it.
Yeah it's generally not a good idea to ask questions than can (even vaguely) imply a problem is the user's fault, and doubly so if you're suggesting it's because they come from [insert cultural background here].
Željko: we call it the hamburger as a joke. It's supposed to represent a list of menu items.
He did however mention she/he wanted a help page but when I asked where that would be discovered he went into radio silence.
I know the design team have been looking to move away from the hamburger icon and after talking with May we reckon it might be useful to run a few A/B tests around it to see if it becomes 'more clickable' if
- a divider is present
- if we try effects on the icon to make it look more tappable
- If it changes to an icon other than the hamburger
The three variables you propose to A/B test are very difficult to create a valid controlled study around. You have to include only users who have never seen the current mobile site, because otherwise you're creating a huge confound by including users who remember the placement and purpose of the current icon. Even if you change the icon entirely for everyone, users remember placement and use it to orient themselves within an interface just as much as icons or other details. I would suggest solving this problem with a more qualitative approach, ala usertesting.com remote tests.
-- Steven Walling, Product Manager https://wikimediafoundation.org/
On Fri, Nov 15, 2013 at 1:24 PM, Jon Robson jrobson@wikimedia.org wrote:
If the A/B test is limited to anonymous users on all pages, then I would expect us to still be able to deduce whether minor changes to the UI encourage clicking (in an audience if 30% of that has never clicked the icon we would still see differences in click through rate in an A/B test as 15% of those would be captured in the A/B test).
You can observe an increase or decrease, but the point is that it's meaningless data, because there is no way to determine that what caused it with any certainty. This means you can run a test and collect data, but you can't answer a question like "Does this version make it easier to find the menu, compared to the old version?"
Compare this to tests mobile has run on newly-registered users. While you can't guarantee that they've all never made an account before, we know through careful analysis that it's very likely that the vast majority of new registrations are in fact new people. So when we do a random 50/50 split of new registrations, we're comparing the behavior of two similar populations of users who have never been exposed to both treatments in an A/B test.
With a random set of readers, you're getting a huge selection of users who might be new, and also many users who have seen some permutation of the site before. With a test like this, there's no way to ensure that a result isn't just do to effects like random exploratory clicking because you introduced something new to people who are used to the old icon. This is a very common problem in A/B testing.
Have we done any of the work we talked about to instrument if in a given session a user clicks on the menu icon? The other thing I'd like to try is to have the site layer "bouncehttps://dl.dropboxusercontent.com/u/30377416/prototypes/multimedia/meda-viewer-desk/mm-viewer-proto.html" (click on the bird picture 2 times) to expose the drawer partially. This would work better if the user could pull the whole content layer to expose the drawer. But even if we can't get that working I still think it would be a useful animation to show there is more content "under" the content.
First step in any of this is getting some instrumentation, without this we're just guessing at how many users it affects. We should at least capture how many expose the menu and how many click on the options within in.
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation
M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Sat, Nov 16, 2013 at 3:24 AM, Steven Walling swalling@wikimedia.orgwrote:
On Fri, Nov 15, 2013 at 1:24 PM, Jon Robson jrobson@wikimedia.org wrote:
If the A/B test is limited to anonymous users on all pages, then I would expect us to still be able to deduce whether minor changes to the UI encourage clicking (in an audience if 30% of that has never clicked the icon we would still see differences in click through rate in an A/B test as 15% of those would be captured in the A/B test).
You can observe an increase or decrease, but the point is that it's meaningless data, because there is no way to determine that what caused it with any certainty. This means you can run a test and collect data, but you can't answer a question like "Does this version make it easier to find the menu, compared to the old version?"
Compare this to tests mobile has run on newly-registered users. While you can't guarantee that they've all never made an account before, we know through careful analysis that it's very likely that the vast majority of new registrations are in fact new people. So when we do a random 50/50 split of new registrations, we're comparing the behavior of two similar populations of users who have never been exposed to both treatments in an A/B test.
With a random set of readers, you're getting a huge selection of users who might be new, and also many users who have seen some permutation of the site before. With a test like this, there's no way to ensure that a result isn't just do to effects like random exploratory clicking because you introduced something new to people who are used to the old icon. This is a very common problem in A/B testing.
-- Steven Walling, Product Manager https://wikimediafoundation.org/
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
On Saturday, November 16, 2013, Jared Zimmerman wrote:
First step in any of this is getting some instrumentation, without this we're just guessing at how many users it affects. We should at least capture how many expose the menu and how many click on the options within in.
Good point. If we can just know what % of viewers (logged in and not) click through now, that's a great start.
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Sat, Nov 16, 2013 at 3:24 AM, Steven Walling <swalling@wikimedia.org<javascript:_e({}, 'cvml', 'swalling@wikimedia.org');>
wrote:
On Fri, Nov 15, 2013 at 1:24 PM, Jon Robson <jrobson@wikimedia.org<javascript:_e({}, 'cvml', 'jrobson@wikimedia.org');>
wrote:
If the A/B test is limited to anonymous users on all pages, then I would expect us to still be able to deduce whether minor changes to the UI encourage clicking (in an audience if 30% of that has never clicked the icon we would still see differences in click through rate in an A/B test as 15% of those would be captured in the A/B test).
You can observe an increase or decrease, but the point is that it's meaningless data, because there is no way to determine that what caused it with any certainty. This means you can run a test and collect data, but you can't answer a question like "Does this version make it easier to find the menu, compared to the old version?"
Compare this to tests mobile has run on newly-registered users. While you can't guarantee that they've all never made an account before, we know through careful analysis that it's very likely that the vast majority of new registrations are in fact new people. So when we do a random 50/50 split of new registrations, we're comparing the behavior of two similar populations of users who have never been exposed to both treatments in an A/B test.
With a random set of readers, you're getting a huge selection of users who might be new, and also many users who have seen some permutation of the site before. With a test like this, there's no way to ensure that a result isn't just do to effects like random exploratory clicking because you introduced something new to people who are