On Mon, Apr 4, 2011 at 22:14, Erik Moeller erik@wikimedia.org wrote:
2011/4/4 Rodan Bury bury.rodan@gmail.com:
As Erik Möller said the qualitative analysis is the user testing with a few dozens of users. This user testing was conducted several times during the development cycle, and it was thorough. The best user testing consist of no more than 30 users, and I can tell the user testing conducted by the Usability Team is high quality and standard.
See also: http://en.wikipedia.org/wiki/Usability_testing#How_many_users_to_test.3F which has links to relevant research.
Note that we did both in-person and remote testing. Remote tests were still focused on US subjects for a variety of reasons (need for reliable connectivity, increasing recruiting and scheduling complexity, etc.). Ultimately I hope chapters can get more involved in on-the-ground user testing in additional locations to surface more culture/language-specific issues.
As for the quantitative analysis, the one made during the beta testing of Vector was detailed. It clearly showed that most users - and especially newbies - preferred Vector over Monobook (retention rates of 70 - 80 % and more).
That's correct. See http://usability.wikimedia.org/wiki/Beta_Feedback_Survey for details which included quite a bit of language-specific analysis and follow-up bugfixes. It was the largest feedback collection regarding a software feature we've ever done and surfaced key issues with specific languages, many of which were resolved.
Now, the Usability Initiative endend in April 2010, soon after the deployment of Vector to all Wikimedia Wikis. The Wikimedia Foundation did not place usability as one of their main priorities
That's not correct. Firstly, we continued deployments and bug fixes after the grant period. As a reminder, full deployment to all projects in all languages was only completed September 1 as the "Phase V" of the roll-out. A lot of this time was spent gathering data and feedback from these remaining projects/languages regarding project or language-specific issues, promoting localization work, etc. Wikimedia is a big and complex beast (or bestiary).
There's also the separate usability initiative concerning multimedia upload, which is ongoing (see http://techblog.wikimedia.org/2011/03/uploadwizard-nearing-1-0/ for the most recent update).
Post-Vector, there were three primary projects that kept the folks who had worked on the original grant-funded project busy:
- After the deployments, the engineering team working on the
initiative asked to be able to spend time on re-architecting the JavaScript/CSS delivery system for MediaWiki, as a necessary precondition for more complex software feature. The result was the ResourceLoader project: http://www.mediawiki.org/wiki/ResourceLoader which is now deployed to all WMF projects.
- The Article Feedback tool. With the Public Policy Initiative we had
taken on the largest project ever to improve content quality in Wikipedia, and Sue asked us to implement a reader-driven article quality assessment tool in order to provide additional measures of success for the project. We also needed article feedback data in order to measure quality change over time on an ongoing basis for other quality-related initiatives. The tool is in production use on a few thousand articles and we're still analyzing the data we're getting before making a final decision on wider deployment. See http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Dat... for our findings to-date.
- MediaWiki 1.17. One of the side-effects of focusing on usability
for so long had been that MediaWiki core code review was neglected and backlogged, much to the dissatisfaction of the volunteer developer community. A lot of joint effort was put into clearing the code review backlog to ensure that we could push out a new MediaWiki release, which happened in February. Balancing strategic projects with code review and integration for volunteer-developed code (which in some cases can be quite complex and labor-intensive) is still very much a work-in-progress.
Nimish specifically also spent a lot of his time helping to support the development and piloting of OpenWebAnalytics as a potential analytics framework to gather better real-time data about what's happening in Wikimedia projects, precisely so we can better measure the effects of the interventions we're making.
The going-forward product development priorities of WMF (not including analytics work) are explained in more detail in the product whitepaper. http://strategy.wikimedia.org/wiki/Product_Whitepaper
Mind you, I'm not at all satisfied with the rate of our progress, but that's generally not because "we're not making X or Y high enough of a priority" or "we suck and we don't know what we're doing", but because we simply don't have enough engineers to do all the development work that it takes to really support a huge and important thing like Wikimedia well. We're continuing to hire engineers in SF and contractors around the world, and we're actively looking into an additional engineering presence in a lower-salary region as part of the 11-12 budgeting process that's currently underway.
why are you so convinced that paying programmers will give a better result, than, let me take an example we all now quite well, paying content editors? could the worrying editor trend be stopped by hiring editors in lower salary regions?
or, to put it in other words, did you ever make a "mediawiki programmer trends study" ? maybe there is something wrong with the software, the process, the usability feedback, or the engineers working on it, that the progress is not satisfying?
rupert.