I'm conducting some usability testing for TWA, as part of an IEG grant and also for a Coursera course on Human-Computer Interaction Design. I thought I'd share the highlights with this list. The second interview has some feedback on the new Getting Started flow. --Jake (Ocaasi)
INTERVIEW ONE
Thoughts on vandalism in articles: "You know it's going to be fixed eventually."
If you edit: "I'm kind of old, using Wikipedia for a long time, not like I can interact with it...It's like an encyclopedia for me. I know I can edit it, but..."
What you'd fix: Wouldn't fix a joke (vandalism) but would fix an error
Sign-up: "Creating an account annoys me"
Userpage: "Creating a userpage is confusing"
First impression: "I could totally get obsessed with this. This could definitely suck me in. This could be bad."
Reliability: "If you look it up [the citation] you find out--it's real. My little sister and brother can use Wikipedia [in school]
Likelihood you'd edit in the next week knowing what you know now (1-10): 8
Could you figure it out on your own: "I'm not that tech savvy. I'm not a computer person. I'm not that confident...I feel like I'm smart enough to edit it. Unsure what would happen if I clicked edit...uncertainty."
-25 year old male from Philadelphia suburbs
INTERVIEW TWO
How you knew you could edit: "Because it's Wikipedia"
Would you edit: "A huge amount of motivation would be needed to edit it" Not a typo, not an incomplete section, not an incorrect fact, not bias about a subject or about a topic of interest. "Only an article that impacted me personally, my company or product...something I'm personally invested in. "I'm generally a lurker."
Expectations: "I would hope that it's easy to edit."
Register an account: "I wouldn't read the sign-up instructions. I'd read as little as necessary." Username was already taken but no warning was given. "Captcha is annoying. I've seen better login flows."
Getting started tour: It doesn't mention the word edit, or 'here's how you can help'. Doesn't feel like an invitation to edit or participate.
Creating a userpage: Clicked the [start userpage link]. "Legal jargon" at the top, looks like "nothing vulgar or obscene and anything you put on Wikipedia is up for grabs for anyone". Notice the 'html' tags when bolding a word. Would prefer a WYSIWY editor. "It feels like it could reassure me a little bit along the way. I'm not confused but I'm more engaged than I want to be. It's forcing me to pay attention and to read directions. I'm mildly disappointed in the interface. It's not the worst thing in the world. It's eh, meh. It's not like I'm going to break up with you, but... I'm not sure how long this is going to last."
Searching for a user's talk page: Search for username leads to articles not userpages. Forced to explore the navigation options. "At this point I'm lost in the abyss. This is stupid."
Leaving a message on a user talk page: "Again I'm becoming confused. I expect a "message button or a send button or a compose button." Tried the [edit] link leading to the full page markup: "I don't feel welcome here. This feels insane." Went back to talk page. "I would like to redesign this thing. 'New section' is not working for me. Put a 'send' button near the username, or at the bottom. I'm comparing this to an email model but I want this to operate like a blog comments section."
Signing a post: "In a million years would never have found the [~~~~] instructions. Tried to sign using markup format with ~~~~ on both sides of the text. Previewed to see that it led to a double signature and removed one. "Having to sign your own posts is ne more thing that you'd say, 'Why hasn't [Wikipedia] figured that out yet."
Article talk pages: "The talk page header makes me feel like something went wrong....I see that this is not a discussion about content but about *improving* the content. But getting that out of this was like pulling teeth". "I have no idea what 'no original research means'... oh, it's giving me instructions, guidelines."
On NPOV policy page: "I would go for the nutshell... 'objectivity' should be the goal.
How feasible would it be to do this without a guide: "Someone would have to pay me to get as far as I did. Makes me thing this is some sort of non-profit thing that doesn't have resources to dive deeply into their UX.
On lack of advertisements: Prefer banner donations to "super annoying ads watching me and tracking me. Non-profit makes it a more "trustworthy 'brand'".
-20 to 30 year old male from New York
Thanks, Jacob -- and welcome to the EE list!
I really appreciate your sharing these insights from your usability testing study.
Studies like these are really helpful for understanding how our users perceive the current experience -- and how to improve it.
I was particularly interested in this comment from the second interview, which matches our own observations about leaving a message on a user talk page:
"Again I'm becoming confused. I expect a "message button or a send button or a compose button." Tried the [edit] link leading to the full page markup: "I don't feel welcome here. This feels insane." Went back to talk page. "I would like to redesign this thing. 'New section' is not working for me. Put a 'send' button near the username, or at the bottom. I'm comparing this to an email model but I want this to operate like a blog comments section."
Amen. This type of user feedback will be very useful for planning our next steps for Flow, in addition to all the research that Brandon, Kim and Oliver are collecting already on this portal: http://www.mediawiki.org/wiki/Flow_Portal
Thanks again for sharing this info, Jacob -- keep us posted !
All the best,
Fabrice
On Apr 5, 2013, at 10:38 AM, Jacob Orlowitz wrote:
I'm conducting some usability testing for TWA, as part of an IEG grant and also for a Coursera course on Human-Computer Interaction Design. I thought I'd share the highlights with this list. The second interview has some feedback on the new Getting Started flow. --Jake (Ocaasi)
INTERVIEW ONE
Thoughts on vandalism in articles: "You know it's going to be fixed eventually."
If you edit: "I'm kind of old, using Wikipedia for a long time, not like I can interact with it...It's like an encyclopedia for me. I know I can edit it, but..."
What you'd fix: Wouldn't fix a joke (vandalism) but would fix an error
Sign-up: "Creating an account annoys me"
Userpage: "Creating a userpage is confusing"
First impression: "I could totally get obsessed with this. This could definitely suck me in. This could be bad."
Reliability: "If you look it up [the citation] you find out--it's real. My little sister and brother can use Wikipedia [in school]
Likelihood you'd edit in the next week knowing what you know now (1-10): 8
Could you figure it out on your own: "I'm not that tech savvy. I'm not a computer person. I'm not that confident...I feel like I'm smart enough to edit it. Unsure what would happen if I clicked edit...uncertainty."
-25 year old male from Philadelphia suburbs
INTERVIEW TWO
How you knew you could edit: "Because it's Wikipedia"
Would you edit: "A huge amount of motivation would be needed to edit it" Not a typo, not an incomplete section, not an incorrect fact, not bias about a subject or about a topic of interest. "Only an article that impacted me personally, my company or product...something I'm personally invested in. "I'm generally a lurker."
Expectations: "I would hope that it's easy to edit."
Register an account: "I wouldn't read the sign-up instructions. I'd read as little as necessary." Username was already taken but no warning was given. "Captcha is annoying. I've seen better login flows."
Getting started tour: It doesn't mention the word edit, or 'here's how you can help'. Doesn't feel like an invitation to edit or participate.
Creating a userpage: Clicked the [start userpage link]. "Legal jargon" at the top, looks like "nothing vulgar or obscene and anything you put on Wikipedia is up for grabs for anyone". Notice the 'html' tags when bolding a word. Would prefer a WYSIWY editor. "It feels like it could reassure me a little bit along the way. I'm not confused but I'm more engaged than I want to be. It's forcing me to pay attention and to read directions. I'm mildly disappointed in the interface. It's not the worst thing in the world. It's eh, meh. It's not like I'm going to break up with you, but... I'm not sure how long this is going to last."
Searching for a user's talk page: Search for username leads to articles not userpages. Forced to explore the navigation options. "At this point I'm lost in the abyss. This is stupid."
Leaving a message on a user talk page: "Again I'm becoming confused. I expect a "message button or a send button or a compose button." Tried the [edit] link leading to the full page markup: "I don't feel welcome here. This feels insane." Went back to talk page. "I would like to redesign this thing. 'New section' is not working for me. Put a 'send' button near the username, or at the bottom. I'm comparing this to an email model but I want this to operate like a blog comments section."
Signing a post: "In a million years would never have found the [~~~~] instructions. Tried to sign using markup format with ~~~~ on both sides of the text. Previewed to see that it led to a double signature and removed one. "Having to sign your own posts is ne more thing that you'd say, 'Why hasn't [Wikipedia] figured that out yet."
Article talk pages: "The talk page header makes me feel like something went wrong....I see that this is not a discussion about content but about *improving* the content. But getting that out of this was like pulling teeth". "I have no idea what 'no original research means'... oh, it's giving me instructions, guidelines."
On NPOV policy page: "I would go for the nutshell... 'objectivity' should be the goal.
How feasible would it be to do this without a guide: "Someone would have to pay me to get as far as I did. Makes me thing this is some sort of non-profit thing that doesn't have resources to dive deeply into their UX.
On lack of advertisements: Prefer banner donations to "super annoying ads watching me and tracking me. Non-profit makes it a more "trustworthy 'brand'".
-20 to 30 year old male from New York
EE mailing list EE@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ee
_______________________________
Fabrice Florin Product Manager, Editor Engagement Wikimedia Foundation
http://en.wikipedia.org/wiki/User:Fabrice_Florin_(WMF)
Donate to keep Wikipedia free: https://donate.wikimedia.org/
Interesting. I've picked out some bits to comment on.
On 05/04/13 18:38, Jacob Orlowitz wrote:
Username was already taken but no warning was given. "Captcha is annoying. I've seen better login flows."
No warning given...? That's a bug. Unfortunately captcha is a necessary compromise... It would be better to be able to get rid of it, but I'm not sure that can be done.
On 05/04/13 18:38, Jacob Orlowitz wrote:
Notice the 'html' tags when bolding a word. Would prefer a WYSIWY editor. "It feels like it could reassure me a little bit along the way. I'm not confused but I'm more engaged than I want to be. It's forcing me to pay attention and to read directions. I'm mildly disappointed in the interface. It's not the worst thing in the world. It's eh, meh. It's not like I'm going to break up with you, but... I'm not sure how long this is going to last."
Could you contact this guy again and show him how to enable VisualEditor (Preferences -> Editing -> Beta features -> Enable VisualEditor), and see how he gets along with that?
On 05/04/13 18:38, Jacob Orlowitz wrote:
Leaving a message on a user talk page: "Again I'm becoming confused. I expect a "message button or a send button or a compose button." Tried the [edit] link leading to the full page markup: "I don't feel welcome here. This feels insane." Went back to talk page. "I would like to redesign this thing. 'New section' is not working for me. Put a 'send' button near the username, or at the bottom. I'm comparing this to an email model but I want this to operate like a blog comments section."
Perhaps 'New discussion'?
On 05/04/13 18:38, Jacob Orlowitz wrote:
Signing a post: "In a million years would never have found the [~~~~] instructions. Tried to sign using markup format with ~~~~ on both sides of the text. Previewed to see that it led to a double signature and removed one. "Having to sign your own posts is ne more thing that you'd say, 'Why hasn't [Wikipedia] figured that out yet."
Until recently my Echo welcome notification message actually said "Please remember to sign any comments on talk pages with 4 tildes (~~~~)." (Admittedly this was because I didn't have any other ideas...) However, it was changed in https://gerrit.wikimedia.org/r/#/c/56057/
Thanks, Alex Monk
On 04/05/2013 02:30 PM, Krenair wrote:
Interesting. I've picked out some bits to comment on.
On 05/04/13 18:38, Jacob Orlowitz wrote:
Username was already taken but no warning was given. "Captcha is annoying. I've seen better login flows."
No warning given...? That's a bug.
Presumably they mean no warning before POSTing to the server.
Jacob, can you tell us the date these interviews were conducted? I notice there's also one that says "Creating an account annoys me". We recently disabled our enhanced account creation (code-named ACUX) in preparation for moving it to core. It would be very useful to know whether these comments were about the old or enhanced version.
Unfortunately captcha is a necessary compromise... It would be better to be able to get rid of it, but I'm not sure that can be done.
We were going to disable it temporarily on English Wikipedia to get an idea of the short-term costs and benefits. It seems this has been shelved for now.
Until recently my Echo welcome notification message actually said "Please remember to sign any comments on talk pages with 4 tildes (~~~~)." (Admittedly this was because I didn't have any other ideas...) However, it was changed in https://gerrit.wikimedia.org/r/#/c/56057/
Yeah, E3 changed it because we wanted to focus more on a welcome per se. Some wikis have bots that will sign automatically for you (https://en.wikipedia.org/wiki/User:SineBot), and medium-term Flow is coming.
Matt Flaschen
On Fri, Apr 5, 2013 at 12:20 PM, Matthew Flaschen mflaschen@wikimedia.orgwrote:
We were going to disable it temporarily on English Wikipedia to get an idea of the short-term costs and benefits. It seems this has been shelved for now.
Yep. This idea is spec'd at https://meta.wikimedia.org/wiki/Research:Account_creation_UX/CAPTCHA. Doing a short removal experiment got support from Chris (Steipp, works on security) but was definitely a touchy subject.
Our alternative idea is to support adding a refresh button to the CAPTCHA wherever it appears (so both account creation and where it appears when saving an edit). This is pretty much already built and working, and you can try it on our Labs instance http://piramido.wmflabs.org/wiki/Main_Page. We just need to make sure it's incorporated well in to our new account creation look being worked on for MediaWiki core.
On 04/05/2013 04:32 PM, Steven Walling wrote:
Our alternative idea is to support adding a refresh button to the CAPTCHA wherever it appears (so both account creation and where it appears when saving an edit). This is pretty much already built and working, and you can try it on our Labs instance http://piramido.wmflabs.org/wiki/Main_Page. We just need to make sure it's incorporated well in to our new account creation look being worked on for MediaWiki core.
Yeah, for simplicity I am planning on updating it once ACUX is in core.
Matt Flaschen
On Fri, Apr 5, 2013 at 10:38 AM, Jacob Orlowitz jorlowitz@gmail.com wrote:
I'm conducting some usability testing for TWA, as part of an IEG grant and also for a Coursera course on Human-Computer Interaction Design. I thought I'd share the highlights with this list. The second interview has some feedback on the new Getting Started flow. --Jake (Ocaasi)
Hey Jake,
These notes are super interesting! One small but very helpful addition would be if you documented a little bit more about who the people being interviewed were. Use of language like "login flows" tells me these might be course mates or people with technical/UX experience? The basic demographics you included are good, but it's also important to get a rough sense of how advanced a Web user the person is.
Thanks again for sharing these. If you want to continue doing usability testing for The Wikipedia Adventure, it might make sense for us to share/document how we're conducting such tests on the Editor Engagement teams.