I've seen some of the results from "agile testing", it seems like they have a tendency to lock in on specific suboptimal solution. What is an acceptable solution on a given limited state.
John
Erik Moeller skrev:
2009/3/25 John at Darkstar vacuum@jeb.no:
Wikipedians should not be used to asses usabillity problems with Wikipedia, this is rule number one if you want to get information about why a newbie has problems with a system.
I'm not sure about your definition of "Wikipedians" above, but the recruiting procedure uses a screening process to recruit _readers_ of Wikipedia with no editing experience. Our goal here is to at the end of the day make improvements to convert more readers to editors.
Ten participants are not nearly enough, they can only give you some clue about the real problem.
There are different philosophies of usability, including a philosophy of agile testing with few test subjects ( http://www.useit.com/alertbox/20000319.html ). There's general agreement that with these kinds of tests, you'll quickly see diminishing returns - adding many more people doesn't actually help you discover many more problems. Moreover, resources are not infinite: finding a good balance in terms of the number of testers allows you to conduct more tests later, with the goal of validating whether the changes you've made actually have had the intended effect.
Erik