On Fri, Nov 15, 2013 at 11:33 AM, Jon Robson <jrobson@wikimedia.org> wrote:
May and I tried to probe a bit more to understand whether it's a
cultural thing (e.g. maybe the hamburger icon is only recognisable in
North America/Western Europe). Alas, he/she then got very defensive
and angry and for some reason didn't want to talk about it.

Yeah it's generally not a good idea to ask questions than can (even vaguely) imply a problem is the user's fault, and doubly so if you're suggesting it's because they come from [insert cultural background here]. 

Željko: we call it the hamburger as a joke. It's supposed to represent a list of menu items. 
He did however mention she/he wanted a help page but when I asked
where that would be discovered he went into radio silence.

I know the design team have been looking to move away from the
hamburger icon and after talking with May we reckon it might be useful
to run a few A/B tests around it to see if it becomes 'more clickable'
* a divider is present
* if we try effects on the icon to make it look more tappable
* If it changes to an icon other than the hamburger

The three variables you propose to A/B test are very difficult to create a valid controlled study around. You have to include only users who have never seen the current mobile site, because otherwise you're creating a huge confound by including users who remember the placement and purpose of the current icon. Even if you change the icon entirely for everyone, users remember placement and use it to orient themselves within an interface just as much as icons or other details. I would suggest solving this problem with a more qualitative approach, ala usertesting.com remote tests. 

Steven Walling,
Product Manager