[Foundation-l] Vector, a year after

[THURNER rupert]

On Mon, Apr 4, 2011 at 22:14, Erik Moeller <erik at wikimedia.org> wrote:
> 2011/4/4 Rodan Bury <bury.rodan at gmail.com>:
>> As Erik Möller said the qualitative analysis is the user testing with a few
>> dozens of users. This user testing was conducted several times during the
>> development cycle, and it was thorough. The best user testing consist of no
>> more than 30 users, and I can tell the user testing conducted by the
>> Usability Team is high quality and standard.
>
> See also:
> http://en.wikipedia.org/wiki/Usability_testing#How_many_users_to_test.3F
> which has links to relevant research.
>
> Note that we did both in-person and remote testing. Remote tests were
> still focused on US subjects for a variety of reasons (need for
> reliable connectivity, increasing recruiting and scheduling
> complexity, etc.). Ultimately I hope chapters can get more involved in
> on-the-ground user testing in additional locations to surface more
> culture/language-specific issues.
>
>> As for the quantitative analysis, the one made during the beta testing of
>> Vector was detailed. It clearly showed that most users - and especially
>> newbies - preferred Vector over Monobook (retention rates of 70 - 80 % and
>> more).
>
> That's correct. See
> http://usability.wikimedia.org/wiki/Beta_Feedback_Survey for details
> which included quite a bit of language-specific analysis and follow-up
> bugfixes. It was the largest feedback collection regarding a software
> feature we've ever done and surfaced key issues with specific
> languages, many of which were resolved.
>
>> Now, the Usability Initiative endend in April 2010, soon after the
>> deployment  of Vector to all Wikimedia Wikis. The Wikimedia Foundation did
>> not place usability as one of their main priorities
>
> That's not correct. Firstly, we continued deployments and bug fixes
> after the grant period. As a reminder, full deployment to all projects
> in all languages was only completed September 1 as the "Phase V" of
> the roll-out. A lot of this time was spent gathering data and feedback
> from these remaining projects/languages regarding project or
> language-specific issues, promoting localization work, etc. Wikimedia
> is a big and complex beast (or bestiary).
>
> There's also the separate usability initiative concerning multimedia
> upload, which is ongoing (see
> http://techblog.wikimedia.org/2011/03/uploadwizard-nearing-1-0/ for
> the most recent update).
>
> Post-Vector, there were three primary projects that kept the folks who
> had worked on the original grant-funded project busy:
>
> 1) After the deployments, the engineering team working on the
> initiative asked to be able to spend time on re-architecting the
> JavaScript/CSS delivery system for MediaWiki, as a necessary
> precondition for more complex software feature. The result was the
> ResourceLoader project:
> http://www.mediawiki.org/wiki/ResourceLoader which is now deployed to
> all WMF projects.
>
> 2) The Article Feedback tool. With the Public Policy Initiative we had
> taken on the largest project ever to improve content quality in
> Wikipedia, and Sue asked us to implement a reader-driven article
> quality assessment tool in order to provide additional measures of
> success for the project. We also needed article feedback data in order
> to measure quality change over time on an ongoing basis for other
> quality-related initiatives. The tool is in production use on a few
> thousand articles and we're still analyzing the data we're getting
> before making a final decision on wider deployment. See
> http://www.mediawiki.org/wiki/Article_feedback/Public_Policy_Pilot/Early_Data
> for our findings to-date.
>
> 3) MediaWiki 1.17. One of the side-effects of focusing on usability
> for so long had been that MediaWiki core code review was neglected and
> backlogged, much to the dissatisfaction of the volunteer developer
> community. A lot of joint effort was put into clearing the code review
> backlog to ensure that we could push out a new MediaWiki release,
> which happened in February. Balancing strategic projects with code
> review and integration for volunteer-developed code (which in some
> cases can be quite complex and labor-intensive) is still very much a
> work-in-progress.
>
> Nimish specifically also spent a lot of his time helping to support
> the development and piloting of OpenWebAnalytics as a potential
> analytics framework to gather better real-time data about what's
> happening in Wikimedia projects, precisely so we can better measure
> the effects of the interventions we're making.
>
> The going-forward product development priorities of WMF (not including
> analytics work) are explained in more detail in the product
> whitepaper. <http://strategy.wikimedia.org/wiki/Product_Whitepaper>
>
> Mind you, I'm not at all satisfied with the rate of our progress, but
> that's generally not because "we're not making X or Y high enough of a
> priority" or "we suck and we don't know what we're doing", but because
> we simply don't have enough engineers to do all the development work
> that it takes to really support a huge and important thing like
> Wikimedia well. We're continuing to hire engineers in SF and
> contractors around the world, and we're actively looking into an
> additional engineering presence in a lower-salary region as part of
> the 11-12 budgeting process that's currently underway.

why are you so convinced that paying programmers will give a better
result, than, let me take an example we all now quite well, paying
content editors? could the worrying editor trend be stopped by hiring
editors in lower salary regions?

or, to put it in other words, did you ever make a "mediawiki
programmer trends study" ? maybe there is something wrong with the
software, the process, the usability feedback, or the engineers
working on it, that the progress is not satisfying?

rupert.