Hi folks,
I just did 3 quick user-tests of the new mobile site layout with some
Android testers. I've got another 3 tests with iPhone users pending.
The mobile testing feature of
usertesting.com is pretty fun & useful,
and I would strongly encourage constant use, especially when we deploy
any significant change to production. It's good for usability testing,
and it can also occasionally surface some nasty bugs that we've not
been able to capture locally or in the wild.
The one downside is that it doesn't seem to give you very detailed
device info, or indeed any device info. All I see in the tester
profile is "Android", which is pretty useless for debugging. I've
asked them if more detailed info is accessible somehow and will let
you know.
Here are my test videos from the Android test. I've added annotations
to each one for some notable moments.
https://accounts.usertesting.com/TestResults.aspx?l6e6u3t7y8i6t3f6
Highlights/summary:
1) It's really frustrating for users when the search essentially just
gives up on anything that's not an exact match. One user repeatedly
tried to do full text search, but was confused by the search button
(the magnifying glass) disappearing in full-screen mode (it gets
replaced by the "X" that clears the search field). She expected the
search button to perform a full text search, and accidentally cleared
her search instead by tapping the "X" button.
2) Although it's not articulated verbally by the testers, 2 of 3 have
problems tapping the tiny "Desktop | Mobile" switcher, especially
given that it's so close to the "Terms of Use" link. We should really
consider increasing the font size here somewhat as they're extremely
hard to tap at the small font size.
3) One tester, using what looks like an early gen Android phone, is
having serious trouble with the section expand/collapse on a
medium-sized page. Here's a clip specifically of that experience:
https://accounts.usertesting.com/ViewClip.aspx?file=iZFnw8acTqMBOpi11hrVow%…
Notice how it collapses the sections waaaay into his reading
experience. Is this a known issue on slower/older phones? If so
perhaps we can degrade a bit more gracefully on those devices.
4) All testers point out the missing infoboxes in the mobile version,
and two explicitly name it as something they want (one tester says
that she was looking up someone's birthdate earlier the same day, and
was not able to find it due to the collapsed infobox). This appears to
be a current bug in production, filed at
https://bugzilla.wikimedia.org/show_bug.cgi?id=36627
5) One tester used swipe gestures to type, which really does not
interact well with our search box. Not sure we can do much about that
one, except that at least with one of his attempts it could have
brought up a "Did you mean" suggestion.
I enjoyed one tester's attempts to use voice input. :) It's also nice
to see the different type of corrections people attempt to do with
different input method.
Hope this is useful; will follow up with iPhone results shortly.
All best,
Erik
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge:
https://wikimediafoundation.org/wiki/Donate