Hi all!
The Wikipedia Usability Initiative conducted a user research study with SF based Bolt Peters in late March to uncover barriers new editors face. We are in the process of completing a full report on our methodology, process and analysis, but wanted to share with you some of the major themes and findings in the meantime....
Some quotes from our participants that illustrate these findings:
“Usually it’s the most information in the easiest spot to access. It always looks very well put together….it boggles my mind how many people can contribute and it still looks like an encyclopedia.” – ‘Galen’
“I like Wikipedia because it’s plain text and nothing flashes” – ‘Claudia’
“Rather than making a mess, I’d rather take some time to figure out how to do it right." (later) "There sure is a lot of stuff to read.” – ‘Dan’
“ [I felt] kind of stupid.” – ‘Galen’
“It’d be nice to have a GUI, so you could see what you’re editing. You’ve made these changes and you’re looking at it, and you don’t know how it’s going to look on the page. It’s a little clumsy to see how it’s going to look.” – ‘Bryan’
“[This is] where I’d give up.” – ‘Shaun’
Check out the full post on the foundation blog: http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/
We would love to hear any initial thoughts, opinions, and reactions. If you have any similar or dissimilar experiences - either personally or in your own work/research, we'd love to hear about that too!
Always on your side, The Usability Team
On Fri, Apr 24, 2009 at 2:08 AM, Parul Vora pvora@wikimedia.org wrote:
Hi all!
The Wikipedia Usability Initiative conducted a user research study with SF based Bolt Peters in late March to uncover barriers new editors face. We are in the process of completing a full report on our methodology, process and analysis, but wanted to share with you some of the major themes and findings in the meantime....
From what I read, the main problem is that new, eager, serious
contributers surrender between our markup and an overwhelming flood of descriptions.
I know a new GUI is being worked on. For the moment I hacked the following JavaScript suggestion:
http://en.wikipedia.org/wiki/User:Magnus_Manske/newbiehelp.js
This adds a "how?" link into the "edit" tab, and launches a floating panel with some extremely general content:
http://en.wikipedia.org/wiki/File:Edit_how.png
Never mind the wording, the color scheme, or important points I missed :-)
If that were added for all anons by default, it might save the willing and able some grief.
Just a thought.
Cheers, Magnus
I know a new GUI is being worked on. For the moment I hacked the following JavaScript suggestion:
http://en.wikipedia.org/wiki/User:Magnus_Manske/newbiehelp.js
This adds a "how?" link into the "edit" tab, and launches a floating panel with some extremely general content:
http://en.wikipedia.org/wiki/File:Edit_how.png
Never mind the wording, the color scheme, or important points I missed :-)
If that were added for all anons by default, it might save the willing and able some grief.
Just a thought.
Great suggestion. One of the repeated sentiment from the study participants was "what is the editing process". This will give a quick overview to anons.
Will you post your idea to our project page? :-) http://usability.wikimedia.org/wiki/Main_Page
Best,
- Naoko
Hi All!
Thanks for all of the feedback, comments, and support. I just wanted to let you know that our full report (including highlight videos!!) is now up our the Usability Initiative's project wiki:
http://usability.wikimedia.org/wiki/UX_and_Usability_Study
- The Usability Team
Parul Vora wrote:
Hi all!
The Wikipedia Usability Initiative conducted a user research study with SF based Bolt Peters in late March to uncover barriers new editors face. We are in the process of completing a full report on our methodology, process and analysis, but wanted to share with you some of the major themes and findings in the meantime....
Some quotes from our participants that illustrate these findings:
“Usually it’s the most information in the easiest spot to access. It always looks very well put together….it boggles my mind how many people can contribute and it still looks like an encyclopedia.” – ‘Galen’
“I like Wikipedia because it’s plain text and nothing flashes” – ‘Claudia’
“Rather than making a mess, I’d rather take some time to figure out how to do it right." (later) "There sure is a lot of stuff to read.” – ‘Dan’
“ [I felt] kind of stupid.” – ‘Galen’
“It’d be nice to have a GUI, so you could see what you’re editing. You’ve made these changes and you’re looking at it, and you don’t know how it’s going to look on the page. It’s a little clumsy to see how it’s going to look.” – ‘Bryan’
“[This is] where I’d give up.” – ‘Shaun’
Check out the full post on the foundation blog: http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/
We would love to hear any initial thoughts, opinions, and reactions. If you have any similar or dissimilar experiences - either personally or in your own work/research, we'd love to hear about that too!
Always on your side, The Usability Team
This usability study is so tiny. I want MediaWiki to be really, really good. Please tell me you guys hope to go large scale with the remote testing setup.
On Thu, May 7, 2009 at 5:41 PM, Parul Vora pvora@wikimedia.org wrote:
Hi All!
Thanks for all of the feedback, comments, and support. I just wanted to let you know that our full report (including highlight videos!!) is now up our the Usability Initiative's project wiki:
http://usability.wikimedia.org/wiki/UX_and_Usability_Study
- The Usability Team
Parul Vora wrote:
Hi all!
The Wikipedia Usability Initiative conducted a user research study with SF based Bolt Peters in late March to uncover barriers new editors face. We are in the process of completing a full report on our methodology, process and analysis, but wanted to share with you some of the major themes and findings in the meantime....
Some quotes from our participants that illustrate these findings:
“Usually it’s the most information in the easiest spot to access. It always looks very well put together….it boggles my mind how many people can contribute and it still looks like an encyclopedia.” – ‘Galen’
“I like Wikipedia because it’s plain text and nothing flashes” – ‘Claudia’
“Rather than making a mess, I’d rather take some time to figure out how to do it right." (later) "There sure is a lot of stuff to read.” – ‘Dan’
“ [I felt] kind of stupid.” – ‘Galen’
“It’d be nice to have a GUI, so you could see what you’re editing. You’ve made these changes and you’re looking at it, and you don’t know how it’s going to look on the page. It’s a little clumsy to see how it’s going to look.” – ‘Bryan’
“[This is] where I’d give up.” – ‘Shaun’
Check out the full post on the foundation blog:
http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/
We would love to hear any initial thoughts, opinions, and reactions. If you have any similar or dissimilar experiences - either personally or in your own work/research, we'd love to hear about that too!
Always on your side, The Usability Team
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Based on these criteria, the 2,500 users that responded to our survey were filtered down to 500 viable subjects based on their answers to these questions. The team, along with B|P, partnered with Davis Recruiting to contact, filter, and screen these 500 participants based on their Wikipedia contribution history, Wikipedia usage patterns, their given reasons for not contributing, and their talkativeness and openness to discuss their thoughts and actions. From 2,500 users, we ended up with 10 study participants and 3-5 waitlisted participants.
You went from 2,500 subjects to just 10? Remote testing allows you to study a virtually unlimited number of participants in a fully natural environment, not some clean room. If you're going to clean up the interface by moving high-utility elements to the areas users tend to look at most, then I'd recommend bringing them into the office. Otherwise, you need to catch them in their regular routine - they perform a Google search, it lands them at Wikipedia. What happens next? Do you feel like you have the answer to this question after your local studies? They see a bit of information is incorrect. How many users *just don't get what Wikipedia is* at that stage? Of those who get it, what happens next? I'm talking about in the course of their day, in the middle of whatever it is they were doing they needed information, how does Wikipedia actually get used? That's a usability test, and you have no clue!
On Thu, May 7, 2009 at 6:22 PM, Brian Brian.Mingus@colorado.edu wrote:
This usability study is so tiny. I want MediaWiki to be really, really good. Please tell me you guys hope to go large scale with the remote testing setup.
On Thu, May 7, 2009 at 5:41 PM, Parul Vora pvora@wikimedia.org wrote:
Hi All!
Thanks for all of the feedback, comments, and support. I just wanted to let you know that our full report (including highlight videos!!) is now up our the Usability Initiative's project wiki:
http://usability.wikimedia.org/wiki/UX_and_Usability_Study
- The Usability Team
Parul Vora wrote:
Hi all!
The Wikipedia Usability Initiative conducted a user research study with SF based Bolt Peters in late March to uncover barriers new editors face. We are in the process of completing a full report on our methodology, process and analysis, but wanted to share with you some of the major themes and findings in the meantime....
Some quotes from our participants that illustrate these findings:
“Usually it’s the most information in the easiest spot to access. It always looks very well put together….it boggles my mind how many people can contribute and it still looks like an encyclopedia.” –
‘Galen’
“I like Wikipedia because it’s plain text and nothing flashes” – ‘Claudia’
“Rather than making a mess, I’d rather take some time to figure out how to do it right." (later) "There sure is a lot of stuff to read.” – ‘Dan’
“ [I felt] kind of stupid.” – ‘Galen’
“It’d be nice to have a GUI, so you could see what you’re editing. You’ve made these changes and you’re looking at it, and you don’t know how it’s going to look on the page. It’s a little clumsy to see how it’s going to look.” – ‘Bryan’
“[This is] where I’d give up.” – ‘Shaun’
Check out the full post on the foundation blog:
http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/
We would love to hear any initial thoughts, opinions, and reactions. If you have any similar or dissimilar experiences - either personally or in your own work/research, we'd love to hear about that too!
Always on your side, The Usability Team
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
2009/5/8 Brian Brian.Mingus@colorado.edu:
You went from 2,500 subjects to just 10?
For a software test, which this mostly was, 5 is enough for excellent results in most cases.
- d.
2009/5/7 Brian Brian.Mingus@colorado.edu:
Based on these criteria, the 2,500 users that responded to our survey were filtered down to 500 viable subjects based on their answers to these questions. The team, along with B|P, partnered with Davis Recruiting to contact, filter, and screen these 500 participants based on their Wikipedia contribution history, Wikipedia usage patterns, their given reasons for not contributing, and their talkativeness and openness to discuss their thoughts and actions. From 2,500 users, we ended up with 10 study participants and 3-5 waitlisted participants.
You went from 2,500 subjects to just 10?
The purpose of a study like this is focused observation of the behavior of individual human beings. As David has pointed out, for any study like this there are laws of diminishing returns, and any serious observation of an individual is time-consuming and costly (raw data is worthless if you can't analyze it). That's why usability gurus like Nielsen suggest "5 is enough" for most tests: http://www.useit.com/alertbox/20000319.html - due to our highly diverse audience, we chose a larger group, and we split between remote and lab testing to compensate for biases of both methods. This has worked well to identify plenty of very obvious usability barriers to focus on.
There are alternative data collection methods such as large scale quantitative testing where the level of individual engagement is limited; those can give you behavioral patterns etc. They can be useful, too, but are an entirely different thing.
This all goes back to how you aim to quantify improvement in usability. These samples sizes are so small that it will be hard (or even impossible) to evaluate your progress based on statistical significance. You've got to prove to us that its really getting better, and doesn't just look prettier.
On Thu, May 7, 2009 at 7:44 PM, Erik Moeller erik@wikimedia.org wrote:
2009/5/7 Brian Brian.Mingus@colorado.edu:
Based on these criteria, the 2,500 users that responded to our survey
were
filtered down to 500 viable subjects based on their answers to these questions. The team, along with B|P, partnered with Davis Recruiting to contact, filter, and screen these 500 participants based on their
Wikipedia
contribution history, Wikipedia usage patterns, their given reasons for
not
contributing, and their talkativeness and openness to discuss their
thoughts
and actions. From 2,500 users, we ended up with 10 study participants
and
3-5 waitlisted participants.
You went from 2,500 subjects to just 10?
The purpose of a study like this is focused observation of the behavior of individual human beings. As David has pointed out, for any study like this there are laws of diminishing returns, and any serious observation of an individual is time-consuming and costly (raw data is worthless if you can't analyze it). That's why usability gurus like Nielsen suggest "5 is enough" for most tests: http://www.useit.com/alertbox/20000319.html - due to our highly diverse audience, we chose a larger group, and we split between remote and lab testing to compensate for biases of both methods. This has worked well to identify plenty of very obvious usability barriers to focus on.
There are alternative data collection methods such as large scale quantitative testing where the level of individual engagement is limited; those can give you behavioral patterns etc. They can be useful, too, but are an entirely different thing. -- Erik Möller Deputy Director, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
2009/5/7 Brian Brian.Mingus@colorado.edu:
This all goes back to how you aim to quantify improvement in usability. These samples sizes are so small that it will be hard (or even impossible) to evaluate your progress based on statistical significance. You've got to prove to us that its really getting better, and doesn't just look prettier.
That's why, in addition to follow-up tests, the usability team is working closely with Erik Zachte to make sure we've got good editing metrics that show whether our changes are actually making a substantial difference in engaging new contributors.
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're going to draw conclusions based on results that simply aren't valid, and you won't know it until the study is over and you didn't make progress.
Careful analysis of site data could allow you to draw some conclusions. I'm curious how you're planning to go about that. Dependent/independent variables?
On Thu, May 7, 2009 at 7:54 PM, Erik Moeller erik@wikimedia.org wrote:
2009/5/7 Brian Brian.Mingus@colorado.edu:
This all goes back to how you aim to quantify improvement in usability. These samples sizes are so small that it will be hard (or even
impossible)
to evaluate your progress based on statistical significance. You've got
to
prove to us that its really getting better, and doesn't just look
prettier.
That's why, in addition to follow-up tests, the usability team is working closely with Erik Zachte to make sure we've got good editing metrics that show whether our changes are actually making a substantial difference in engaging new contributors. -- Erik Möller Deputy Director, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
On Thu, May 7, 2009 at 8:58 PM, Brian Brian.Mingus@colorado.edu wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're going to draw conclusions based on results that simply aren't valid, and you won't know it until the study is over and you didn't make progress.
Careful analysis of site data could allow you to draw some conclusions. I'm curious how you're planning to go about that. Dependent/independent variables?
If five subjects, chosen at random, all have the same problem, then with 95% confidence you can predict that at least half of the population will report having this problem.
This kind of work generally focuses on BIG problems, and you don't need a huge sample to identify some of the most common issues. In things like UI development it would be surprising if there weren't complaints reported by most of the subjects. You may overlook some other problems, but when coming up with a list of common problems to work on, I would say that 15 subjects is plenty.
-Robert Rohde
Sounds easy. I wonder why this "study" doesn't mention a p value. The grant must not have been large enough to fund someone with any experience using R, or god forbid, a pencil.
On Thu, May 7, 2009 at 10:32 PM, Robert Rohde rarohde@gmail.com wrote:
On Thu, May 7, 2009 at 8:58 PM, Brian Brian.Mingus@colorado.edu wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're
going
to draw conclusions based on results that simply aren't valid, and you
won't
know it until the study is over and you didn't make progress.
Careful analysis of site data could allow you to draw some conclusions.
I'm
curious how you're planning to go about that. Dependent/independent variables?
If five subjects, chosen at random, all have the same problem, then with 95% confidence you can predict that at least half of the population will report having this problem.
This kind of work generally focuses on BIG problems, and you don't need a huge sample to identify some of the most common issues. In things like UI development it would be surprising if there weren't complaints reported by most of the subjects. You may overlook some other problems, but when coming up with a list of common problems to work on, I would say that 15 subjects is plenty.
-Robert Rohde
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Ok, I'll agree that the motiviations and size of this pilot study are reasonable. Then I'd just like to know how much money was spent getting these answers. If you're not planning to measure the subjects scientifically and you just want to figure out what the big issues are then the premise of the lab itself comes into question.
On Thu, May 7, 2009 at 10:32 PM, Robert Rohde rarohde@gmail.com wrote:
On Thu, May 7, 2009 at 8:58 PM, Brian Brian.Mingus@colorado.edu wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're
going
to draw conclusions based on results that simply aren't valid, and you
won't
know it until the study is over and you didn't make progress.
Careful analysis of site data could allow you to draw some conclusions.
I'm
curious how you're planning to go about that. Dependent/independent variables?
If five subjects, chosen at random, all have the same problem, then with 95% confidence you can predict that at least half of the population will report having this problem.
This kind of work generally focuses on BIG problems, and you don't need a huge sample to identify some of the most common issues. In things like UI development it would be surprising if there weren't complaints reported by most of the subjects. You may overlook some other problems, but when coming up with a list of common problems to work on, I would say that 15 subjects is plenty.
-Robert Rohde
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
I've been told in private messages that I'm not allowed to ask these questions and that my style is excessively rude. Pardon me for continuing, I think they are important issues. If I come across rude to you just imagine me in person where I am much more likable in conversation. Otherwise, get over it!
On Fri, May 8, 2009 at 8:28 AM, Brian Brian.Mingus@colorado.edu wrote:
Ok, I'll agree that the motiviations and size of this pilot study are reasonable. Then I'd just like to know how much money was spent getting these answers. If you're not planning to measure the subjects scientifically and you just want to figure out what the big issues are then the premise of the lab itself comes into question.
On Thu, May 7, 2009 at 10:32 PM, Robert Rohde rarohde@gmail.com wrote:
On Thu, May 7, 2009 at 8:58 PM, Brian Brian.Mingus@colorado.edu wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're
going
to draw conclusions based on results that simply aren't valid, and you
won't
know it until the study is over and you didn't make progress.
Careful analysis of site data could allow you to draw some conclusions.
I'm
curious how you're planning to go about that. Dependent/independent variables?
If five subjects, chosen at random, all have the same problem, then with 95% confidence you can predict that at least half of the population will report having this problem.
This kind of work generally focuses on BIG problems, and you don't need a huge sample to identify some of the most common issues. In things like UI development it would be surprising if there weren't complaints reported by most of the subjects. You may overlook some other problems, but when coming up with a list of common problems to work on, I would say that 15 subjects is plenty.
-Robert Rohde
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
On Thu, May 7, 2009 at 11:58 PM, Brian Brian.Mingus@colorado.edu wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're going to draw conclusions based on results that simply aren't valid, and you won't know it until the study is over and you didn't make progress.
Careful analysis of site data could allow you to draw some conclusions. I'm curious how you're planning to go about that. Dependent/independent variables?
An exercise in statistical thinking: when everyone or almost everyone cites problem X, how many people does it take to reach statistical significance that X is a problem worth addressing? Even if the results are a statistical fluke and in reality only 20% of new users run into trouble with problem X, that's still a problem worth addressing.
The fact that so many of the 15 people had the same problems, and those problems also align with common sense, is a strong indication that the study has found some things worth fixing.
There is more than one way to come to reliable conclusions. Any time I see someone invoking "the [singular] scientific method", as if there is only one and it is set in stone and universally agreed upon by all rational people, I have trouble taking them seriously. See [[Talk:Scientific method]].
-Sage (User:Ragesoss)
I will have no part in your efforts to redefine the scientific method on its talk page.
On Thu, May 7, 2009 at 10:35 PM, Sage Ross <ragesoss+wikipedia@gmail.comragesoss%2Bwikipedia@gmail.com
wrote:
On Thu, May 7, 2009 at 11:58 PM, Brian Brian.Mingus@colorado.edu wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're
going
to draw conclusions based on results that simply aren't valid, and you
won't
know it until the study is over and you didn't make progress.
Careful analysis of site data could allow you to draw some conclusions.
I'm
curious how you're planning to go about that. Dependent/independent variables?
An exercise in statistical thinking: when everyone or almost everyone cites problem X, how many people does it take to reach statistical significance that X is a problem worth addressing? Even if the results are a statistical fluke and in reality only 20% of new users run into trouble with problem X, that's still a problem worth addressing.
The fact that so many of the 15 people had the same problems, and those problems also align with common sense, is a strong indication that the study has found some things worth fixing.
There is more than one way to come to reliable conclusions. Any time I see someone invoking "the [singular] scientific method", as if there is only one and it is set in stone and universally agreed upon by all rational people, I have trouble taking them seriously. See [[Talk:Scientific method]].
-Sage (User:Ragesoss)
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
2009/5/8 Brian Brian.Mingus@colorado.edu:
I will have no part in your efforts to redefine the scientific method on its talk page.
Fortunately you don't need to. People who have put far more effort into the subject than you are ever likely to do so have pretty much shot apart the idea of a single hard scientific method that scientists actually use.
I don't recall making such a claim. You seem very interested in making it seem as if I had, however.
On Fri, May 8, 2009 at 4:25 AM, geni geniice@gmail.com wrote:
2009/5/8 Brian Brian.Mingus@colorado.edu:
I will have no part in your efforts to redefine the scientific method on
its
talk page.
Fortunately you don't need to. People who have put far more effort into the subject than you are ever likely to do so have pretty much shot apart the idea of a single hard scientific method that scientists actually use.
-- geni
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Brian wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're going to draw conclusions based on results that simply aren't valid, and you won't know it until the study is over and you didn't make progress.
I would caution people unfamiliar with the scientific literature in the area from making overbroad about fields outside their area of expertise. There are pros and cons of quantitative and qualitative studies in human-computer interaction (HCI), and in many cases qualitative studies give more useful information, especially for fairly early stage design questions. In particular, it's quite easy to get statistically rigorous but in-practice useless data from quantitative studies. Those are very good when you have precisely one variable you want to measure (or maybe a few), but in many HCI studies, especially with open-ended interface design questions (like "make Wikipedia better") there are thousands of possible variables, and a major purpose of a study is to better understand the concerns of the domain and figure out what the relevant variables even *are*. Quantitative studies are much more useful when you've fixed your interface but have a handful of final decisions to make---do we want a particular feature to have slight variant A or B, while everything else stays fixed.
-Mark
You don't have to be an expert (more formally defined as someone with ten years of experience in a field) to spot unscientific thinking. I don't think you're an expert either so maybe you should just leave expertise out of it. More to the point, the question in my mind is, how much money did the Foundation waste on this "study" that you can't draw valid, generalizable conclusions from? The distinction between quantitative and qualitative is for naught - you can, and should, quantify what you mean when making qualitative assessments. Especially if you will use them to bolster your arguments about how to spend millions of dollars of donated money.
On Fri, May 8, 2009 at 3:51 AM, Delirium delirium@hackish.org wrote:
Brian wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists. It's unscientific thinking and it's going cause to you waste money. You're
going
to draw conclusions based on results that simply aren't valid, and you
won't
know it until the study is over and you didn't make progress.
I would caution people unfamiliar with the scientific literature in the area from making overbroad about fields outside their area of expertise. There are pros and cons of quantitative and qualitative studies in human-computer interaction (HCI), and in many cases qualitative studies give more useful information, especially for fairly early stage design questions. In particular, it's quite easy to get statistically rigorous but in-practice useless data from quantitative studies. Those are very good when you have precisely one variable you want to measure (or maybe a few), but in many HCI studies, especially with open-ended interface design questions (like "make Wikipedia better") there are thousands of possible variables, and a major purpose of a study is to better understand the concerns of the domain and figure out what the relevant variables even *are*. Quantitative studies are much more useful when you've fixed your interface but have a handful of final decisions to make---do we want a particular feature to have slight variant A or B, while everything else stays fixed.
-Mark
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
On Fri, May 8, 2009 at 10:10 AM, Brian Brian.Mingus@colorado.edu wrote:
You don't have to be an expert (more formally defined as someone with ten years of experience in a field) to spot unscientific thinking. I don't think you're an expert either so maybe you should just leave expertise out of it. More to the point, the question in my mind is, how much money did the Foundation waste on this "study" that you can't draw valid, generalizable conclusions from? The distinction between quantitative and qualitative is for naught - you can, and should, quantify what you mean when making qualitative assessments. Especially if you will use them to bolster your arguments about how to spend millions of dollars of donated money.
Honestly, you as much as admit that you have no idea what you're talking about. You could at least be a little more collegial and a little less accusatory if you are looking for good faith responses to your complaints.
Nathan
Really, I admit that? Where.
On Fri, May 8, 2009 at 8:30 AM, Nathan nawrich@gmail.com wrote:
On Fri, May 8, 2009 at 10:10 AM, Brian Brian.Mingus@colorado.edu wrote:
You don't have to be an expert (more formally defined as someone with ten years of experience in a field) to spot unscientific thinking. I don't think you're an expert either so maybe you should just leave expertise out of
it.
More to the point, the question in my mind is, how much money did the Foundation waste on this "study" that you can't draw valid, generalizable conclusions from? The distinction between quantitative and qualitative is for naught - you can, and should, quantify what you mean when making qualitative assessments. Especially if you will use them to bolster your arguments about how to spend millions of dollars of donated money.
Honestly, you as much as admit that you have no idea what you're talking about. You could at least be a little more collegial and a little less accusatory if you are looking for good faith responses to your complaints.
Nathan _______________________________________________ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
On Thu, May 7, 2009 at 11:58 PM, Brian Brian.Mingus@colorado.edu wrote:
Quite frankly the advice that you should only use five subjects makes no sense. The appeal to Nielsen's authority is not going to work on me or anyone else who understands why the scientific method exists.
Experience shows that most people end up being very similar when it comes to usability. Most problems show up repeatedly even with groups of five people. If you run the tests on a hundred people, you're going to get a somewhat more accurate picture, but not enough to justify the extra expense. It's much better to run a five-person study, assume that any objections raised by (say) at least three are representative, fix those, and run another few five-person studies on the fixed software for the same cost.
You don't need large sample sizes if something is regular enough. You only need high sample sizes if the object of your study is variable enough to require it. That's usually the case in pharmacological studies, for instance, but that doesn't mean it's true everywhere. If you have a tiny standard deviation, then a study of five people could provide very clear conclusions. It all depends on your data.
2009/5/8 Brian Brian.Mingus@colorado.edu:
This usability study is so tiny. I want MediaWiki to be really, really good. Please tell me you guys hope to go large scale with the remote testing setup.
Nit just mediawiki. Looks like we need to improve the paths through the help namespace.
That may be true. This study does not allow you to draw that conclusion, however.
On Thu, May 7, 2009 at 6:36 PM, geni geniice@gmail.com wrote:
2009/5/8 Brian Brian.Mingus@colorado.edu:
This usability study is so tiny. I want MediaWiki to be really, really
good.
Please tell me you guys hope to go large scale with the remote testing setup.
Nit just mediawiki. Looks like we need to improve the paths through the help namespace.
-- geni
foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
El 5/7/09 5:36 PM, geni escribió:
2009/5/8 BrianBrian.Mingus@colorado.edu:
This usability study is so tiny. I want MediaWiki to be really, really good. Please tell me you guys hope to go large scale with the remote testing setup.
Nit just mediawiki. Looks like we need to improve the paths through the help namespace.
The primary target on the help pages is to make key pieces like the markup "cheat-sheet" available quickly, easily, and universally.
Wikia's got a cute little editing-help-sidebar version of this which we've considered adapting or taking as a model for that purpose -- it gets you some important stuff without taking you out of your editing environment.
Style guides, licensing recommendations, etc are another level of help page which needs more consideration, but remain of secondary importance.
-- brion
Parul Vora wrote:
Thanks for all of the feedback, comments, and support. I just wanted to let you know that our full report (including highlight videos!!) is now up our the Usability Initiative's project wiki:
It may be too late for this now, but when I read that "The cheat sheet was the only item in the help section that led to a subjects successful edit.", I wanted to know how well would people edit if they had the printed cheat sheet. Maybe if you have more tests in future you could do this - it will tell us exactly what is the value of the printed promotional material.
I am wary of this: "Users often missed the ‘edit’ buttons next to each section, clicking on ‘edit this page’ all the way at the top." In my experience, users do exactly the opposite, and I have seen new users who know how to edit sections asking how to edit top section; some Wikipedias (f.e. ruwiki) have even added [edit] link to top of the article that mimics section edit links. What could be the cause of this discrepancy?
Would it be possible to get some conclusions and recommendations from this study so that Wikipedia's interface could be improved even before a full WYSIVYG interface is developed? (Because, from past experience, that will take years.) For example, one obvious thing is that editing help should be moved to top right of the edit box. (Maybe it could also pop next to the editing box instead of in a new window?) Another obvious thing, "Create a new article" link should be added somewhere. Yet another, toolbar images should be replaced with text.
2009/5/8 Nikola Smolenski smolensk@eunet.yu:
I am wary of this: "Users often missed the ‘edit’ buttons next to each section, clicking on ‘edit this page’ all the way at the top." In my experience, users do exactly the opposite, and I have seen new users who know how to edit sections asking how to edit top section; some Wikipedias (f.e. ruwiki) have even added [edit] link to top of the article that mimics section edit links. What could be the cause of this discrepancy?
Yes. This is also quite common question sent to OTRS. People quite often ask "I know how to edit the section, but I don't know how to edit the top part of the article". By the way: maybe it is good idea to ask OTRS English team to mark for one week or month all E-mail asking for help in editing and than to make some sumarization of it.
On Fri, May 8, 2009 at 1:35 AM, Nikola Smolenski smolensk@eunet.yu wrote:
Parul Vora wrote:
Thanks for all of the feedback, comments, and support. I just wanted to let you know that our full report (including highlight videos!!) is now up our the Usability Initiative's project wiki:
It may be too late for this now, but when I read that "The cheat sheet was the only item in the help section that led to a subjects successful edit.", I wanted to know how well would people edit if they had the printed cheat sheet. Maybe if you have more tests in future you could do this - it will tell us exactly what is the value of the printed promotional material.
I am wary of this: "Users often missed the ‘edit’ buttons next to each section, clicking on ‘edit this page’ all the way at the top." In my experience, users do exactly the opposite, and I have seen new users who know how to edit sections asking how to edit top section; some Wikipedias (f.e. ruwiki) have even added [edit] link to top of the article that mimics section edit links. What could be the cause of this discrepancy?
About this: on en:wp, at least, under user preferences/gadgets, users can turn this on themselves by clicking the "Add an [edit] link for the lead section of a page" box. Is there any particular reason not to turn this on by default for everyone? Could be (one small) problem (temporarily) solved.
Regarding this whole discussion in general, and comments made by Brian (who I respect quite a lot) -- as I understand it, "usability" and HCI are something of an inexact science. It seems like several approaches and studies would be helpful, to see if different methods come to more or less the same conclusions about what's broken, or not.
There are also a number of world-respected HCI experts that are interested in this community that we could probably draw on for input; see past Wikimania speaker lists.
-- Phoebe
El 5/8/09 9:21 PM, phoebe ayers escribió:
About this: on en:wp, at least, under user preferences/gadgets, users can turn this on themselves by clicking the "Add an [edit] link for the lead section of a page" box. Is there any particular reason not to turn this on by default for everyone? Could be (one small) problem (temporarily) solved.
We disabled the section-0 edit link a couple years ago because it interfered with the floating infoboxes and images which appear at the top of most pages.
Finding a way to put it back that's reliable and interferes with neither floating boxes nor the various badge icons that a lot of templated put in above the line is on the task list. :) Once it's back permanently we can dump the temporary gadgets.
-- brion
On Fri, May 8, 2009 at 9:45 AM, Brion Vibber brion@wikimedia.org wrote:
El 5/8/09 9:21 PM, phoebe ayers escribió:
About this: on en:wp, at least, under user preferences/gadgets, users can turn this on themselves by clicking the "Add an [edit] link for the lead section of a page" box. Is there any particular reason not to turn this on by default for everyone? Could be (one small) problem (temporarily) solved.
We disabled the section-0 edit link a couple years ago because it interfered with the floating infoboxes and images which appear at the top of most pages.
Finding a way to put it back that's reliable and interferes with neither floating boxes nor the various badge icons that a lot of templated put in above the line is on the task list. :) Once it's back permanently we can dump the temporary gadgets.
-- brion
Oh right, infoboxes! Sorry, for a minute there I got all 2002 and forgot they existed. Sigh.
-- Phoebe
wikimedia-l@lists.wikimedia.org