Hi folks,
I just did 3 quick user-tests of the new mobile site layout with some Android testers. I've got another 3 tests with iPhone users pending. The mobile testing feature of usertesting.com is pretty fun & useful, and I would strongly encourage constant use, especially when we deploy any significant change to production. It's good for usability testing, and it can also occasionally surface some nasty bugs that we've not been able to capture locally or in the wild.
The one downside is that it doesn't seem to give you very detailed device info, or indeed any device info. All I see in the tester profile is "Android", which is pretty useless for debugging. I've asked them if more detailed info is accessible somehow and will let you know.
Here are my test videos from the Android test. I've added annotations to each one for some notable moments.
https://accounts.usertesting.com/TestResults.aspx?l6e6u3t7y8i6t3f6
Highlights/summary:
1) It's really frustrating for users when the search essentially just gives up on anything that's not an exact match. One user repeatedly tried to do full text search, but was confused by the search button (the magnifying glass) disappearing in full-screen mode (it gets replaced by the "X" that clears the search field). She expected the search button to perform a full text search, and accidentally cleared her search instead by tapping the "X" button.
2) Although it's not articulated verbally by the testers, 2 of 3 have problems tapping the tiny "Desktop | Mobile" switcher, especially given that it's so close to the "Terms of Use" link. We should really consider increasing the font size here somewhat as they're extremely hard to tap at the small font size.
3) One tester, using what looks like an early gen Android phone, is having serious trouble with the section expand/collapse on a medium-sized page. Here's a clip specifically of that experience:
https://accounts.usertesting.com/ViewClip.aspx?file=iZFnw8acTqMBOpi11hrVow%3...
Notice how it collapses the sections waaaay into his reading experience. Is this a known issue on slower/older phones? If so perhaps we can degrade a bit more gracefully on those devices.
4) All testers point out the missing infoboxes in the mobile version, and two explicitly name it as something they want (one tester says that she was looking up someone's birthdate earlier the same day, and was not able to find it due to the collapsed infobox). This appears to be a current bug in production, filed at https://bugzilla.wikimedia.org/show_bug.cgi?id=36627
5) One tester used swipe gestures to type, which really does not interact well with our search box. Not sure we can do much about that one, except that at least with one of his attempts it could have brought up a "Did you mean" suggestion.
I enjoyed one tester's attempts to use voice input. :) It's also nice to see the different type of corrections people attempt to do with different input method.
Hope this is useful; will follow up with iPhone results shortly.
All best, Erik
This is great and I agree we should do much more of this to validate. I look forward to the iphone results!
Firstly the info boxes should never have disappeared. This somehow sneaked into a big commit I made and I've reverted it in https://gerrit.wikimedia.org/r/6915 - hopefully we can get this deployed later today as there have been a few people complaining about this on Twitter.
* I notice that user Bradz1988 and license2ill both expect pressing enter to perform a full text search rather than the top result (we should review https://bugzilla.wikimedia.org/show_bug.cgi?id=35528) * It highlights that search really should be full text to be more useful 'finnish heavy metal music' didn't return any results :-( * The search seems to makes kayaker feel stupid - this is bad. She expects a search button and doesn't understand that her search results are showing below. Also on clicking search we don't help her by highlighting all the text - this evidence goes against https://bugzilla.wikimedia.org/show_bug.cgi?id=19935 (I've updated the ticket) * We also make license2ill feel like an idiot (he even says so) - as Erik suggests we should be thinking more about providing did you mean functionality * Switching from desktop to mobile is not as easy to locate. * A scroll to top function is much needed (license2ill)
The collapsing sections problem is interesting. His connection goes slow later on in the video which is the cause of the problem. The javascript currently runs at the end of the page load as I personally believe content is the most important thing to serve the user first. This works fine on smaller pages but on larger pages as shown here can be disastrous. I think the real fix here is to keep the page size down and to dynamically load sections on demand. I've opened a bug - https://bugzilla.wikimedia.org/show_bug.cgi?id=36633
On Tue, May 8, 2012 at 9:06 AM, Erik Moeller erik@wikimedia.org wrote:
Hi folks,
I just did 3 quick user-tests of the new mobile site layout with some Android testers. I've got another 3 tests with iPhone users pending. The mobile testing feature of usertesting.com is pretty fun & useful, and I would strongly encourage constant use, especially when we deploy any significant change to production. It's good for usability testing, and it can also occasionally surface some nasty bugs that we've not been able to capture locally or in the wild.
The one downside is that it doesn't seem to give you very detailed device info, or indeed any device info. All I see in the tester profile is "Android", which is pretty useless for debugging. I've asked them if more detailed info is accessible somehow and will let you know.
Here are my test videos from the Android test. I've added annotations to each one for some notable moments.
https://accounts.usertesting.com/TestResults.aspx?l6e6u3t7y8i6t3f6
Highlights/summary:
- It's really frustrating for users when the search essentially just
gives up on anything that's not an exact match. One user repeatedly tried to do full text search, but was confused by the search button (the magnifying glass) disappearing in full-screen mode (it gets replaced by the "X" that clears the search field). She expected the search button to perform a full text search, and accidentally cleared her search instead by tapping the "X" button.
- Although it's not articulated verbally by the testers, 2 of 3 have
problems tapping the tiny "Desktop | Mobile" switcher, especially given that it's so close to the "Terms of Use" link. We should really consider increasing the font size here somewhat as they're extremely hard to tap at the small font size.
- One tester, using what looks like an early gen Android phone, is
having serious trouble with the section expand/collapse on a medium-sized page. Here's a clip specifically of that experience:
https://accounts.usertesting.com/ViewClip.aspx?file=iZFnw8acTqMBOpi11hrVow%3...
Notice how it collapses the sections waaaay into his reading experience. Is this a known issue on slower/older phones? If so perhaps we can degrade a bit more gracefully on those devices.
- All testers point out the missing infoboxes in the mobile version,
and two explicitly name it as something they want (one tester says that she was looking up someone's birthdate earlier the same day, and was not able to find it due to the collapsed infobox). This appears to be a current bug in production, filed at https://bugzilla.wikimedia.org/show_bug.cgi?id=36627
- One tester used swipe gestures to type, which really does not
interact well with our search box. Not sure we can do much about that one, except that at least with one of his attempts it could have brought up a "Did you mean" suggestion.
I enjoyed one tester's attempts to use voice input. :) It's also nice to see the different type of corrections people attempt to do with different input method.
Hope this is useful; will follow up with iPhone results shortly.
All best, Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Kudos and many thanks to Erik for constructing and running these highly realistic tests, which are a huge wake-up call.
Each video is an encyclopedia of learning.
I only wish we had run tests like these long ago, or at minimum on each beta release.
Some high-level points, and then evenings and weekends of study to analyze the specific pain points.
1. We fail, in a sense, when the user says the desktop view is easier or better. That conclusion seems pretty consistent.
2. Search needs work, lots of it. People think it should work like Google, and it doesn't. But even what we do, which is search suggestions, is not that friendly, and we still do not have full-text search on the mobile site, or search in article text on the site or apps.
(By the way, "Did you mean" was implemented at the SF Hackathon, but made it only into the apps. Also, the discussion about full-text search that occurred during the release of the Android app needs to be continued and finished, but at least there is a form of full-text search in the apps.)
There are a few specific usability issues that may be a factor in the big drop in search when the beta went to production on May 1 - but all of the issues mentioned above are huge general issues.
Android user 2, license2ill, had excellent comments about the current search.
3. We have a lot to optimize in terms of in-article navigation and download, some of which Jon is working on, and which will be improved in the new UI.
Much, much more to come.
If anyone wants to login to take a look, here you go:
https://www.usertesting.com/dashboard user: rfarrand@wikimedia.org pw: wikimediapenguin!
Phil
On Tue, May 8, 2012 at 2:54 AM, Jon Robson jdlrobson@gmail.com wrote:
This is great and I agree we should do much more of this to validate. I look forward to the iphone results!
Firstly the info boxes should never have disappeared. This somehow sneaked into a big commit I made and I've reverted it in https://gerrit.wikimedia.org/r/6915 - hopefully we can get this deployed later today as there have been a few people complaining about this on Twitter.
- I notice that user Bradz1988 and license2ill both expect pressing
enter to perform a full text search rather than the top result (we should review https://bugzilla.wikimedia.org/show_bug.cgi?id=35528)
- It highlights that search really should be full text to be more
useful 'finnish heavy metal music' didn't return any results :-(
- The search seems to makes kayaker feel stupid - this is bad. She
expects a search button and doesn't understand that her search results are showing below. Also on clicking search we don't help her by highlighting all the text - this evidence goes against https://bugzilla.wikimedia.org/show_bug.cgi?id=19935 (I've updated the ticket)
- We also make license2ill feel like an idiot (he even says so) - as
Erik suggests we should be thinking more about providing did you mean functionality
- Switching from desktop to mobile is not as easy to locate.
- A scroll to top function is much needed (license2ill)
The collapsing sections problem is interesting. His connection goes slow later on in the video which is the cause of the problem. The javascript currently runs at the end of the page load as I personally believe content is the most important thing to serve the user first. This works fine on smaller pages but on larger pages as shown here can be disastrous. I think the real fix here is to keep the page size down and to dynamically load sections on demand. I've opened a bug - https://bugzilla.wikimedia.org/show_bug.cgi?id=36633
On Tue, May 8, 2012 at 9:06 AM, Erik Moeller erik@wikimedia.org wrote:
Hi folks,
I just did 3 quick user-tests of the new mobile site layout with some Android testers. I've got another 3 tests with iPhone users pending. The mobile testing feature of usertesting.com is pretty fun & useful, and I would strongly encourage constant use, especially when we deploy any significant change to production. It's good for usability testing, and it can also occasionally surface some nasty bugs that we've not been able to capture locally or in the wild.
The one downside is that it doesn't seem to give you very detailed device info, or indeed any device info. All I see in the tester profile is "Android", which is pretty useless for debugging. I've asked them if more detailed info is accessible somehow and will let you know.
Here are my test videos from the Android test. I've added annotations to each one for some notable moments.
https://accounts.usertesting.com/TestResults.aspx?l6e6u3t7y8i6t3f6
Highlights/summary:
- It's really frustrating for users when the search essentially just
gives up on anything that's not an exact match. One user repeatedly tried to do full text search, but was confused by the search button (the magnifying glass) disappearing in full-screen mode (it gets replaced by the "X" that clears the search field). She expected the search button to perform a full text search, and accidentally cleared her search instead by tapping the "X" button.
- Although it's not articulated verbally by the testers, 2 of 3 have
problems tapping the tiny "Desktop | Mobile" switcher, especially given that it's so close to the "Terms of Use" link. We should really consider increasing the font size here somewhat as they're extremely hard to tap at the small font size.
- One tester, using what looks like an early gen Android phone, is
having serious trouble with the section expand/collapse on a medium-sized page. Here's a clip specifically of that experience:
https://accounts.usertesting.com/ViewClip.aspx?file=iZFnw8acTqMBOpi11hrVow%3...
Notice how it collapses the sections waaaay into his reading experience. Is this a known issue on slower/older phones? If so perhaps we can degrade a bit more gracefully on those devices.
- All testers point out the missing infoboxes in the mobile version,
and two explicitly name it as something they want (one tester says that she was looking up someone's birthdate earlier the same day, and was not able to find it due to the collapsed infobox). This appears to be a current bug in production, filed at https://bugzilla.wikimedia.org/show_bug.cgi?id=36627
- One tester used swipe gestures to type, which really does not
interact well with our search box. Not sure we can do much about that one, except that at least with one of his attempts it could have brought up a "Did you mean" suggestion.
I enjoyed one tester's attempts to use voice input. :) It's also nice to see the different type of corrections people attempt to do with different input method.
Hope this is useful; will follow up with iPhone results shortly.
All best, Erik -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson http://jonrobson.me.uk @rakugojon
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Tue, May 8, 2012 at 11:22 AM, Philip Chang pchang@wikimedia.org wrote:
If anyone wants to login to take a look, here you go:
Erm, please don't share passwords with a publicly archived mailing list.
I've changed the password; contact me offlist for access.
As I told you before, usertesting.com allows easy publication of videos without requiring login. The links I shared work without being logged in.
On Tue, May 8, 2012 at 1:06 AM, Erik Moeller erik@wikimedia.org wrote:
Here are my test videos from the Android test. I've added annotations to each one for some notable moments.
And the iPhone ones: https://accounts.usertesting.com/TestResults.aspx?p4n1n6v8t4h3i4e5
The consistent pattern here is the same frustration with hitting "No results" in the search. Will watch more closely tonight to see if I spot any other major issues that have not been mentioned already.
Let me side-nap this thread to note that tapping a footnote reference (in article text) on the Android app doesn't take me to the footnote. Is this a known issue? If not, I'll go and file it with full details.
Asaf
On Tue, May 8, 2012 at 3:47 PM, Erik Moeller erik@wikimedia.org wrote:
On Tue, May 8, 2012 at 1:06 AM, Erik Moeller erik@wikimedia.org wrote:
Here are my test videos from the Android test. I've added annotations to each one for some notable moments.
And the iPhone ones: https://accounts.usertesting.com/TestResults.aspx?p4n1n6v8t4h3i4e5
The consistent pattern here is the same frustration with hitting "No results" in the search. Will watch more closely tonight to see if I spot any other major issues that have not been mentioned already. -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Tue, May 8, 2012 at 11:56 PM, Asaf Bartov abartov@wikimedia.org wrote:
Let me side-nap this thread to note that tapping a footnote reference (in article text) on the Android app doesn't take me to the footnote. Is this a known issue? If not, I'll go and file it with full details.
This should be fixed now - it was also a bug on the mobile site https://bugzilla.wikimedia.org/show_bug.cgi?id=36618
These videos are good - they allow you to see users of different abilities levels using the mobile versions of Wikipedia - maybe even something to consider if we roll out changes to the desktop site. A few questions,
1) Can you set specific demographics such as location though? Looking at the iPhone one, it sounds like a couple are in American and one in the UK, but having a global testing or different countries each time might be a good idea.
2) Can we use this for the Wikipedia app, the one that is downloaded from the App Market / App Store - I think this will be very useful there.
3) I want to add annotations to videos, such as for https://accounts.usertesting.com/ViewVideo.aspx?file=VDKgNg5FETo%3d# I can comment on the competency and confidence of the user using their iPhone and also how they seem to be missing the point of looking at the desktop site on their *iPhone *not their desktop! this would be good for clearer instructions in the future. I can't seem to be able add annotations though to this, do I need to be logged in for this?
On Tue, May 8, 2012 at 6:03 PM, Thehelpfulone thehelpfulonewiki@gmail.com wrote:
- Can you set specific demographics such as location though? Looking at
the iPhone one, it sounds like a couple are in American and one in the UK, but having a global testing or different countries each time might be a good idea.
You can specify demographics, but in terms of region, usertesting.com is currently limited to the US, UK, and Canada.
- Can we use this for the Wikipedia app, the one that is downloaded from
the App Market / App Store - I think this will be very useful there.
Since there's no magic involved (people are literally filming their phones), you can ask them to do pretty much anything, including installation of an app.
- I want to add annotations to videos, such as
for https://accounts.usertesting.com/ViewVideo.aspx?file=VDKgNg5FETo%3d#%C2%A0I can comment on the competency and confidence of the user using their iPhone and also how they seem to be missing the point of looking at the desktop site on their iPhone not their desktop! this would be good for clearer instructions in the future. I can't seem to be able add annotations though to this, do I need to be logged in for this?
It requires logging in. You can however download any of the videos, encode them with ffmpeg2theora and upload them to Commons. We have permission to do so under CC-BY per their terms of use:
https://www.usertesting.com/privacy-policy
On 9 May 2012 02:28, Erik Moeller erik@wikimedia.org wrote:
You can specify demographics, but in terms of region, usertesting.com is currently limited to the US, UK, and Canada.
Okay, in that case it would be good to have users with different competency levels using their phones to see how they differ in their views and usage of the app/mobile site.
Since there's no magic involved (people are literally filming their phones), you can ask them to do pretty much anything, including installation of an app.
Then this would be a good idea to get people that have never used the Wikipedia app before to film trying it out so that we understand what initial impressions during first use and difficulties faced, how easy it is for them to see all the features of the app, if they can figure out what the Read it Later service is, asking them to compare their experience to the mobile site...
It requires logging in. You can however download any of the videos, encode them with ffmpeg2theora and upload them to Commons. We have permission to do so under CC-BY per their terms of use:
Okay, I can do this for all 6 videos later today, if they're >100MB I'll try the UploadWizard again or request a server side upload. Do you have a user name/password that I can use to add the annotations?
Files have been uploaded, you can see them in https://commons.wikimedia.org/wiki/Category:Wikimedia_Mobile_videos.
Thanks so much!
Just to give a little more detail on user selection in usertesting.com, it seems that aside from very general demographic preferences, basically age range and gender, it is possible to specify highest rated testers and favorite testers.
By not selecting any preferences, it seems we can get a diverse sampling of skill levels, and perhaps even more importantly, the testers are not sampled from our "normal" audience. The net effect is more like that of a focus group, but with the key difference that users are in their natural setting using their own devices (along the lines of country-specific user studies the Foundation has been performing in recent years).
Thanks for being helpful (here and in mobile development).
Phil
On Wed, May 9, 2012 at 8:03 AM, Thehelpfulone thehelpfulonewiki@gmail.comwrote:
Files have been uploaded, you can see them in https://commons.wikimedia.org/wiki/Category:Wikimedia_Mobile_videos. -- Thehelpfulone
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
We discovered a regression regarding full-text search on the site.
A quick take on the blank search results: there is a blank screen at the beginning, which is off-putting, and at the end of typing (when no article matches the string entered). Ordinarily, hitting search on the keyboard would invoke full-text search, and spelling errors would invoke "Did you mean."
We need to fix full-text search urgently, and then make it more intuitive, as well as apply "Did you mean." But we also need to make the types of search more obvious, in terms of their intended behavior. Otherwise people will keep thinking this is "not as easy as Google search."
During the Android app development, Yuvi suggested a way of blending type-ahead suggestions and full-text search, but the initial take was to separate them. This will take more thinking through and clearly should be part of the new UI project.
Phil
PS: Thanks for the suggestion about useragentstring.com.
On Tue, May 8, 2012 at 3:47 PM, Erik Moeller erik@wikimedia.org wrote:
On Tue, May 8, 2012 at 1:06 AM, Erik Moeller erik@wikimedia.org wrote:
Here are my test videos from the Android test. I've added annotations to each one for some notable moments.
And the iPhone ones: https://accounts.usertesting.com/TestResults.aspx?p4n1n6v8t4h3i4e5
The consistent pattern here is the same frustration with hitting "No results" in the search. Will watch more closely tonight to see if I spot any other major issues that have not been mentioned already. -- Erik Möller VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Tue, May 8, 2012 at 1:06 AM, Erik Moeller erik@wikimedia.org wrote:
The one downside is that it doesn't seem to give you very detailed device info, or indeed any device info.
The recommended approach is to have testers visit useragentstring.com as part of the test. This is in fact a default task (number 4), so let's give that a try next time. A bit clunky but should help surface some useful debugging info.