Kudos and many thanks to Erik for constructing and running these highly realistic tests, which are a huge wake-up call.

Each video is an encyclopedia of learning.

I only wish we had run tests like these long ago, or at minimum on each beta release.

Some high-level points, and then evenings and weekends of study to analyze the specific pain points.

1. We fail, in a sense, when the user says the desktop view is easier or better. That conclusion seems pretty consistent.

2. Search needs work, lots of it. People think it should work like Google, and it doesn't. But even what we do, which is search suggestions, is not that friendly, and we still do not have full-text search on the mobile site, or search in article text on the site or apps.

(By the way, "Did you mean" was implemented at the SF Hackathon, but made it only into the apps. Also, the discussion about full-text search that occurred during the release of the Android app needs to be continued and finished, but at least there is a form of full-text search in the apps.)

There are a few specific usability issues that may be a factor in the big drop in search when the beta went to production on May 1 - but all of the issues mentioned above are huge general issues.

Android user 2, license2ill, had excellent comments about the current search.

3. We have a lot to optimize in terms of in-article navigation and download, some of which Jon is working on, and which will be improved in the new UI. 

Much, much more to come.

If anyone wants to login to take a look, here you go:

user: rfarrand@wikimedia.org
pw: wikimediapenguin!


On Tue, May 8, 2012 at 2:54 AM, Jon Robson <jdlrobson@gmail.com> wrote:
This is great and I agree we should do much more of this to validate.
I look forward to the iphone results!

Firstly the info boxes should never have disappeared. This somehow
sneaked into a big commit I made and I've reverted it in
https://gerrit.wikimedia.org/r/6915 - hopefully we can get this
deployed later today as there have been a few people complaining about
this on Twitter.

* I notice that user Bradz1988 and  license2ill  both expect pressing
enter to perform a full text search rather than the top result (we
should review https://bugzilla.wikimedia.org/show_bug.cgi?id=35528)
* It highlights that search really should be full text to be more
useful 'finnish heavy metal music' didn't return any results :-(
* The search seems to makes kayaker feel stupid - this is bad. She
expects a search button and doesn't understand that her search results
are showing below. Also on clicking search we don't help her by
highlighting all the text - this evidence goes against
https://bugzilla.wikimedia.org/show_bug.cgi?id=19935 (I've updated the
* We also make  license2ill  feel like an idiot (he even says so) - as
Erik suggests we should be thinking more about providing did you mean
* Switching from desktop to mobile is not as easy to locate.
* A scroll to top function is much needed (license2ill)

The collapsing sections problem is interesting. His connection goes
slow later on in the video which is the cause of the problem. The
javascript currently runs at the end of the page load as I personally
believe content is the most important thing to serve the user first.
This works fine on smaller pages but on larger pages as shown here can
be disastrous. I think the real fix here is to keep the page size down
and to dynamically load sections on demand. I've opened a bug -

On Tue, May 8, 2012 at 9:06 AM, Erik Moeller <erik@wikimedia.org> wrote:
> Hi folks,
> I just did 3 quick user-tests of the new mobile site layout with some
> Android testers. I've got another 3 tests with iPhone users pending.
> The mobile testing feature of usertesting.com is pretty fun & useful,
> and I would strongly encourage constant use, especially when we deploy
> any significant change to production. It's good for usability testing,
> and it can also occasionally surface some nasty bugs that we've not
> been able to capture locally or in the wild.
> The one downside is that it doesn't seem to give you very detailed
> device info, or indeed any device info. All I see in the tester
> profile is "Android", which is pretty useless for debugging. I've
> asked them if more detailed info is accessible somehow and will let
> you know.
> Here are my test videos from the Android test. I've added annotations
> to each one for some notable moments.
> https://accounts.usertesting.com/TestResults.aspx?l6e6u3t7y8i6t3f6
> Highlights/summary:
> 1) It's really frustrating for users when the search essentially just
> gives up on anything that's not an exact match. One user repeatedly
> tried to do full text search, but was confused by the search button
> (the magnifying glass) disappearing in full-screen mode (it gets
> replaced by the "X" that clears the search field). She expected the
> search button to perform a full text search, and accidentally cleared
> her search instead by tapping the "X" button.
> 2) Although it's not articulated verbally by the testers, 2 of 3 have
> problems tapping the tiny "Desktop | Mobile" switcher, especially
> given that it's so close to the "Terms of Use" link. We should really
> consider increasing the font size here somewhat as they're extremely
> hard to tap at the small font size.
> 3) One tester, using what looks like an early gen Android phone, is
> having serious trouble with the section expand/collapse on a
> medium-sized page. Here's a clip specifically of that experience:
> https://accounts.usertesting.com/ViewClip.aspx?file=iZFnw8acTqMBOpi11hrVow%3d%3d
> Notice how it collapses the sections waaaay into his reading
> experience. Is this a known issue on slower/older phones? If so
> perhaps we can degrade a bit more gracefully on those devices.
> 4) All testers point out the missing infoboxes in the mobile version,
> and two explicitly name it as something they want (one tester says
> that she was looking up someone's birthdate earlier the same day, and
> was not able to find it due to the collapsed infobox). This appears to
> be a current bug in production, filed at
> https://bugzilla.wikimedia.org/show_bug.cgi?id=36627
> 5) One tester used swipe gestures to type, which really does not
> interact well with our search box. Not sure we can do much about that
> one, except that at least with one of his attempts it could have
> brought up a "Did you mean" suggestion.
> I enjoyed one tester's attempts to use voice input. :) It's also nice
> to see the different type of corrections people attempt to do with
> different input method.
> Hope this is useful; will follow up with iPhone results shortly.
> All best,
> Erik
> --
> Erik Möller
> VP of Engineering and Product Development, Wikimedia Foundation
> Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
> _______________________________________________
> Mobile-l mailing list
> Mobile-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/mobile-l

Jon Robson

Mobile-l mailing list

Phil Inje Chang
Product Manager, Mobile
Wikimedia Foundation
415-812-0854 m
415-882-7982 x 6810