From: David Gerard dgerard@gmail.com Date: Thu, Mar 6, 2014 at 9:44 PM Subject: Re: [Wikitech-l] Should MediaWiki CSS prefer non-free fonts? To: Wikimedia developers wikitech-l@lists.wikimedia.org
(Veering off topic: So what does WMF use for a usability lab, anyway?)
...not sure what Kaldari did. In this case, he may have simply sat down with the UX designers and done a test in person.
We do not have a usability testing lab on-site in San Francisco, and typically prefer to do remote usability tests. Either we do this "manually" via sending out a survey,[1] and then running a Google Hangout which we record for later. This is good since it is guided by the person who wrote the test script, so they can adapt to what the user is doing/failing to do. It takes a lot more leg work though.
More often, we write a testing script and use usertesting.com, which is more automated remote testing and is $35/test (this is really cheap since the going US rate for an in-person test is something like a $50 Amazon gift card). The service uses people from all over the English-speaking world who have a variety of levels of technical expertise, and the tests are recorded for viewing after they're completed.[2]
The UX team is actually in the process of hiring a UX researcher, so expect to hear more about this kind of qualitative research soon.
Steven
1. We recently did this kind of recruitment and testing for article drafts work. https://www.mediawiki.org/wiki/Draft_namespace/Usability_testing and the /Results subpage 2. This kind of testing is something we used during the account creation redesign https://www.mediawiki.org/wiki/Account_creation_user_experience/User_testing
_______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 6 March 2014 23:47, Steven Walling steven.walling@gmail.com wrote:
more automated remote testing and is $35/test (this is really cheap since the going US rate for an in-person test is something like a $50 Amazon gift card).
off-topic on off-topic: Offer swag instead. Wikipedia branded stuff is presently uncommon enough to *delight* people. I remember doing a usability test for Ubuntu and accepting some stickers and a £2 USB stick rather than a £40 cheque ... I could tell it was a £2 USB because it stopped working 6 months later.
Anyway. Work the swag angle. Puzzle globes. People LOVE that stuff.
- d.
I just wanted to add that in the past, as many people know, we tried a few different kinds of testing and even hired a usability testing firm to help us. We conducted research in a lab here in SF and also did some remote testing, compensating participants with gift cards.
We learned that lab testing is very expensive, complicated and slow. It has its own unique filtering qualities that prevent certain kinds of people from participating and encourage others. Participants being in a foreign environment, using someone else's computer and being run through tasks with a giant 2-way mirror behind their back and cameras rolling might distort behavior a bit.
Remote testing done with a facilitator and screen-sharing (like what Steven is talking about with Google Hangout) is still time consuming, but far cheaper and easier than lab testing and can be done on shorter notice. It filters out less tech-savvy people or those who use alternative or legacy devices like phones, tablets or older computers. It's interesting that it allows people to use a computer they are already familiar with, but it may not be relevant to the test.
Remote testing done using usertesting.com is the cheapest and easiest, but even further filters out less tech-savvy people.
I believe, from lots of first-hand experience and some research on the subject, that anytime you can get at least 5 users in front of a product and run them through well written tasks you are going to reveal about 80% of the problems. Getting fancy with the methodology usually only affects the final 20%.
I'm really looking forward to having a UX testing person on staff who can facilitate more testing. I find it very valuable and would like to do more in the future.
- Trevor
On Thu, Mar 6, 2014 at 4:06 PM, David Gerard dgerard@gmail.com wrote:
On 6 March 2014 23:47, Steven Walling steven.walling@gmail.com wrote:
more automated remote testing and is $35/test (this is really cheap since the going US rate for an in-person test is something like a $50 Amazon
gift
card).
off-topic on off-topic: Offer swag instead. Wikipedia branded stuff is presently uncommon enough to *delight* people. I remember doing a usability test for Ubuntu and accepting some stickers and a £2 USB stick rather than a £40 cheque ... I could tell it was a £2 USB because it stopped working 6 months later.
Anyway. Work the swag angle. Puzzle globes. People LOVE that stuff.
- d.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 7 March 2014 00:17, Trevor Parscal tparscal@wikimedia.org wrote:
I believe, from lots of first-hand experience and some research on the subject, that anytime you can get at least 5 users in front of a product and run them through well written tasks you are going to reveal about 80% of the problems. Getting fancy with the methodology usually only affects the final 20%.
I have frequently seen the claim that a usable usability test can be done with five test subjects. I suppose there's betas and mailing ists and wiki forums and other such yelling shops for the other 20% of the problems.
- d.
On Mar 7, 2014 12:48 AM, "Steven Walling" steven.walling@gmail.com wrote:
From: David Gerard dgerard@gmail.com Date: Thu, Mar 6, 2014 at 9:44 PM Subject: Re: [Wikitech-l] Should MediaWiki CSS prefer non-free fonts? To: Wikimedia developers wikitech-l@lists.wikimedia.org
(Veering off topic: So what does WMF use for a usability lab, anyway?)
...not sure what Kaldari did. In this case, he may have simply sat down with the UX designers and done a test in person.
We do not have a usability testing lab on-site in San Francisco, and typically prefer to do remote usability tests. Either we do this
"manually"
via sending out a survey,[1] and then running a Google Hangout which we record for later. This is good since it is guided by the person who wrote the test script, so they can adapt to what the user is doing/failing to
do.
It takes a lot more leg work though.
More often, we write a testing script and use usertesting.com, which is more automated remote testing and is $35/test (this is really cheap since the going US rate for an in-person test is something like a $50 Amazon
gift
card). The service uses people from all over the English-speaking world
Is there anything in place yet for the non English speaking world? If not, is it planned to have any, and if so, is that a priority?
--Martijn.
who
have a variety of levels of technical expertise, and the tests are
recorded
for viewing after they're completed.[2]
The UX team is actually in the process of hiring a UX researcher, so
expect
to hear more about this kind of qualitative research soon.
Steven
- We recently did this kind of recruitment and testing for article drafts
work. https://www.mediawiki.org/wiki/Draft_namespace/Usability_testing and the /Results subpage 2. This kind of testing is something we used during the account creation redesign
https://www.mediawiki.org/wiki/Account_creation_user_experience/User_testing
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hoi, Steven do I understand correctly that there is no user testing except in English ? Thanks, GerardM
On 7 March 2014 00:47, Steven Walling steven.walling@gmail.com wrote:
From: David Gerard dgerard@gmail.com Date: Thu, Mar 6, 2014 at 9:44 PM Subject: Re: [Wikitech-l] Should MediaWiki CSS prefer non-free fonts? To: Wikimedia developers wikitech-l@lists.wikimedia.org
(Veering off topic: So what does WMF use for a usability lab, anyway?)
...not sure what Kaldari did. In this case, he may have simply sat down with the UX designers and done a test in person.
We do not have a usability testing lab on-site in San Francisco, and typically prefer to do remote usability tests. Either we do this "manually" via sending out a survey,[1] and then running a Google Hangout which we record for later. This is good since it is guided by the person who wrote the test script, so they can adapt to what the user is doing/failing to do. It takes a lot more leg work though.
More often, we write a testing script and use usertesting.com, which is more automated remote testing and is $35/test (this is really cheap since the going US rate for an in-person test is something like a $50 Amazon gift card). The service uses people from all over the English-speaking world who have a variety of levels of technical expertise, and the tests are recorded for viewing after they're completed.[2]
The UX team is actually in the process of hiring a UX researcher, so expect to hear more about this kind of qualitative research soon.
Steven
- We recently did this kind of recruitment and testing for article drafts
work. https://www.mediawiki.org/wiki/Draft_namespace/Usability_testing and the /Results subpage 2. This kind of testing is something we used during the account creation redesign
https://www.mediawiki.org/wiki/Account_creation_user_experience/User_testing
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Fri, Mar 7, 2014 at 9:29 AM, Gerard Meijssen gerard.meijssen@gmail.comwrote:
Steven do I understand correctly that there is no user testing except in English ?
You can only do usability testing (i.e. sit down with a person and listen to them talk, or do it remotely) if you understand their language. Otherwise you're just listening to someone give feedback you can't understand.
Someone multilingual like Pau may be able to do tests in languages like Spanish or Catalan, which I believe he might have in the past. But we almost exclusively test in English because it's our universal working language, and we're usually not designing specifically for non-English projects (at least in my work anyway).
Steven
This doesn't have to be just the language that the person conducting the test knows. It goes even further. I remember at least one case where the user could understand English, but couldn't speak it, so he listened to Pau, but replied mostly in Russian, and later I translated the recording.
On a more general and practical note, the test conductor and the user need to converse in a language that they know, but the user interface of the feature being tested can be in another language, which the test conductor doesn't know. It's supposed to be very easy for the test conductor to identify the user interface elements even if they are not labeled in his language.
Needless to say, *any* UI feature should be tested not only in English, and it's perfectly feasible.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2014-03-07 12:19 GMT+02:00 Steven Walling steven.walling@gmail.com:
On Fri, Mar 7, 2014 at 9:29 AM, Gerard Meijssen gerard.meijssen@gmail.comwrote:
Steven do I understand correctly that there is no user testing except in English ?
You can only do usability testing (i.e. sit down with a person and listen to them talk, or do it remotely) if you understand their language. Otherwise you're just listening to someone give feedback you can't understand.
Someone multilingual like Pau may be able to do tests in languages like Spanish or Catalan, which I believe he might have in the past. But we almost exclusively test in English because it's our universal working language, and we're usually not designing specifically for non-English projects (at least in my work anyway).
Steven _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org