One of the important components of the usability initiative is to conduct multiple rounds of usability tests. The plan is to conduct at least three rounds of tests for qualitative usability evaluation over the span of twelve months, i) the initial evaluation, ii) the progress evaluation, and iii) the final evaluation. The initial usability test is scheduled on March 24, 25th and 26th. In-person lab tests are conducted in San Francisco at the first two days, and remote tests will be conducted on the third day.
As a preparation for the initial usability test, we incorporated the recruiting tool into English Wikipedia's site notice. You might have encountered site notice inviting for the participation. The target audience of testers are Wikipedia readers who have little or no experience in editing the Wikipedia articles. The banner is displayed within the range of 1:400 to 1:100 page views, and it will continue till early next week.
We look forward to learning from the usability tests and sharing the result with you.
Thanks.
Naoko ... on behalf of the usability team.
A bit late, I'm catching up on emails. :)
On Fri, Mar 6, 2009 at 08:51, Naoko Komura nkomura@wikimedia.org wrote:
Are there any plans to have usability tests in other languages than English and other "cultures" than "North-American"? It seems to me these two factors actually would play quite a role in the way people interact with the software and probably should be taken into account.
Thanx,
Delphine
2009/3/10 Delphine Ménard notafishz@gmail.com:
Presumable the "remote tests" mentioned will be non-North American, otherwise I'm not sure the point of remote tests (is there really that big a difference between the SF area and the rest of the continent?). It does seem it will be all English speaking, though, which is unfortunate.
Thomas Dalton wrote:
Hi, Delphine and Thomas.
The Stanton Usability grant has a focus on English Wikipedia, but the usability team's intention and wish is to expand the scope so that all languages can benefit the usability improvement. Having said that , we only can allocate the research budget in conducting the usability test for English Wikipedia. The site notice you saw for the past few days were for the lab tests which take place in San Francisco on March 24th and 25th, so we only recruited people who can physically be in the lab in San Francisco on those days. For remote testing, there will be no regional constraints. However as Thomas pointed out, it will be limited for English Wikipedia. Once we establish the methodology of remote testing, we would like to share the methodology and perhaps involve chapters in funding the usability tests for other languages or find a way to conduct usability tests among volunteers.
- Naoko
On Tue, Mar 10, 2009 at 7:02 PM, Naoko Komura nkomura@wikimedia.org wrote:
Purely out of interest: Was this a restriction of the Stanton grant or was it a decision by the WMF (usability team)?
Michael
Michael Bimmler wrote:
Hi, Michael.
The restriction is from the Stanton grant.
- Naoko
On Mar 10, 2009, at 1:36 PM, Naoko Komura wrote:
Is there a restriction on the use of the research? I mean, is there any reason why the eventual outcomes can't be shared with the other projects, or is the propogation of that research confined to the english 'pedia? I'm fairly certain I know the answer (knowledge for ALL!) but would like to confirm that.
Philippe
philippe wrote:
Hi, Philippe.
Just so that the audience of the list does not lose context of the original thread, I would like to reiterate the email thread between Michael and myself.
Michael's question was as follows; "Purely out of interest: Was this (implying the use of the usability research budget) a restriction of the Stanton grant or was it a decision by the WMF (usability team)?"
And my answer was; "The restriction is from the Stanton grant."
You are raising a different question with regards to the use of research results. There is no restriction in the use or sharing the test results publicly as long as we respect and protect privacy of test participants. We are video taping the lab test and plan to make it publicly available, as long as we have consents from the participants.
I hope my answer helps to clear your concerns.
- Naoko
On Tue, Mar 10, 2009 at 7:36 PM, Naoko Komura nkomura@wikimedia.org wrote:
Speaking of which: Has the WMF considered publishing the document (contract? terms?) which details the exact cover, purpose, conditions and obligations of the Stanton grant? I don't know whether the Stanton Foundation wants to keep this document confidential, but then, I don't really see what could be confidential about the terms of grant now that all the negotiations have finished.
I think it would be a nice example of transparency to publish (ie. upload on meta or somewhere) this and similar documents, as long as this does not threaten the obtaining of future grants.
Michael
2009/3/10 Michael Bimmler mbimmler@gmail.com:
I've been thinking about this as well. There is a strong tradition of confidentiality in these processes but I really would like to get some approved grant applications out there for discussion and review. I plan to raise the issue with some of our funders, especially where we have long-standing relationships, to feel it out a bit.
I am comfortable sharing some guiding principles which we include in every tech grant application (begin quote):
* “Release early, release often.” This key concept, which is common to both open source development and agile software development methodologies, simply means that changes are made iteratively, openly, and transparently, encouraging continuing review by peers, volunteer developers and user groups. Designs and tests are revisited and revised as iterative milestones are accomplished. * Openness to outside collaboration. All development will happen in the Wikimedia Foundation's public code repository. Progress reports will be published in blogs, and outside contributions will be explicitly encouraged. * Building on the work of others. An environmental scan will identify existing open source solutions which can be refined and built upon to meet defined requirements. As above, external contributors will be invited to collaborate. * Building on the knowledge of others. When possible, we will try to hire developers with existing MediaWiki development experience. But, to the extent that this is not possible, we will facilitate face-to-face workshops with paid and volunteer developers to share and document knowledge about the MediaWiki architecture. * Readiness for internationalization. Code contributions, whether from volunteer developers or paid developers, will be made in line with Wikimedia's internationalization architecture to allow user interface messages to be localized into all languages. * Consultation of stakeholders. While our goal is to encourage and broaden participation in Wikipedia, we will strive to do so in consultation with the existing community of contributors, in order to help foster their understanding and acceptance of changes to the user interface.
(end quote)
As you can see, internationalization is one of our core principles which we consider to be non-negotiable. (And Gerard will rightly point out that we can do a better job supporting i18n initiatives like BetaWiki.) That said, in this particular case, the funder's interest is specifically to increase the number of contributors to the English Wikipedia, and that places some limitations on what we can use the grant funds for. We are committed to sharing recommendations, usability videos, methodologies, etc. - Naoko will post more on this as the process plays out.
While there are definitely strong cultural components to usability, many of the core issues we're going to try to resolve are shared across all languages. For example, complex pages accumulate syntax creep everywhere, not just in the English Wikipedia. Roughly speaking, I think 80% of the work we're going to do is going to be generally useful, and 20% is going to be specific to the English Wikipedia. Hopefully we can contribute to processes both in communities and chapters to think more systematically about addressing usability issues that are specific to a certain language or culture.
Howdy. (adding wikien-l folks to this thread. my apology for not including wikien-l with my initial email.)
The usability study has started today as scheduled. The usability team is monitoring the interviews and how ten test participants interact with Wikipedia when they are asked to edit an article at the lab facility in San Francisco today and tomorrow. The remote usability study on Thursday (March 26 PDT) will be done remotely, which means we recruit participants from Wikipedia through the site notice, and connect with them through web conferencing. Therefore the site notice for recruitment will appear again on Thursday. We expect to compile the results in a few weeks and the findings with you.
Naoko Komura Program Manager, Wikimedia Foundation
Naoko Komura wrote:
On Tue, Mar 24, 2009 at 9:28 PM, Naoko Komura nkomura@wikimedia.org wrote:
This is brilliant - you could automate the testing and reporting in addition to having a virtually unlimited number of subjects.
On Tue, Mar 24, 2009 at 9:15 PM, Brian Brian.Mingus@colorado.edu wrote:
It lacks some feedback from tests done in a lab - the video cameras recording eye motion and focus, what someone says to themselves or others, and user mouse patterns, are important data.
But more data is always good. If you can't get someone to meet you in the lab, testing via web conferencing has been shown to be good data.
Dear Aphaia,
I'm really excited to get the result of this study since one of the common input that I got from newbie contributors either online or during workshops are that it's harder for them to write Wikipedia than to write blogs :) Though I still don't get it why they think that way ;)
Good luck and success for the effort.
-- Ivan Lanin | Wikimedia Indonesia | http://wikimedia.or.id Sent from my BlackBerry®
-----Original Message----- From: Naoko Komura nkomura@wikimedia.org
Date: Tue, 24 Mar 2009 20:28:45 To: Wikimedia Foundation Mailing Listfoundation-l@lists.wikimedia.org; wikien-l@lists.wikimedia.org Subject: [Foundation-l] Usability study in progress
Howdy. (adding wikien-l folks to this thread. my apology for not including wikien-l with my initial email.)
The usability study has started today as scheduled. The usability team is monitoring the interviews and how ten test participants interact with Wikipedia when they are asked to edit an article at the lab facility in San Francisco today and tomorrow. The remote usability study on Thursday (March 26 PDT) will be done remotely, which means we recruit participants from Wikipedia through the site notice, and connect with them through web conferencing. Therefore the site notice for recruitment will appear again on Thursday. We expect to compile the results in a few weeks and the findings with you.
Naoko Komura Program Manager, Wikimedia Foundation
Naoko Komura wrote:
Hoi, Naoko is Naoko and Aphaia is someone else..
http://wikimediafoundation.org/wiki/File:Naoko_Komura_December_2008.jpg http://upload.wikimedia.org/wikipedia/commons/e/e6/Benutzer_Aphaia.jpg
Thanks, GerardM
2009/3/25 Ivan Lanin ivan.lanin@wikimedia.or.id
Wikipedians should not be used to asses usabillity problems with Wikipedia, this is rule number one if you want to get information about why a newbie has problems with a system. A typical wikipedian is simply not a valid newbie. Ten participants are not nearly enough, they can only give you some clue about the real problem.
John
Naoko Komura skrev:
Ten is a low number indeed, however, if those people are indeed 'typical users' instead of Wikipedians and you given them a few specific tasks (say, searching for an article on a topic they are interested in and editing it to add some information) you will probably encounter lots of problems soon enough.
On a different note: i'm not sure if this has been discussed before but will the usability study also take uploading media on Commons in account? Editing text is one thing, but adding media (and hence, using Commons) is almost as common and could also use *lots* of work on increasing usability.
-- Hay
On Wed, Mar 25, 2009 at 11:55 AM, John at Darkstar vacuum@jeb.no wrote:
One additional note, in Norway a lot of the newspapers used a layout like Monobook (sort of) but has lately dismissed the solution in favor of radically much simpler designs. Especially the list of links in a left bar has been abandoned in favor of more directed approaches with horizontal menus in the top of the pages. Some of the newspapers has reported instantaneous increase in click rates.
It is thought provocative as most of the left bar is for wikipediams, and therefore is usable to only a small percentile of the total users (in page views). The rest of the users need navigational aids for the main space.
John
Hay (Husky) skrev:
Hoi, A newspaper wants people to read. We want very much that readers consider the option to edit. The approach is therefore different. When I goto Wikipedia as a reader, I might be enabled to change my role and consequently get a different layout. Thanks, GerardM
2009/3/25 John at Darkstar vacuum@jeb.no
according to the Usability home page "initial focus on English Wikipedia, eventually this research and development will be implemented across all languages and possibly to other WikiMedia projectshttp://en.wikipedia.org/wiki/Wikimedia_projects#Wikimedia_projects" so i would guess initially not.
regards
mark
On Wed, Mar 25, 2009 at 11:48 AM, Hay (Husky) huskyr@gmail.com wrote:
On Wed, Mar 25, 2009 at 7:48 AM, Hay (Husky) huskyr@gmail.com wrote:
As far as I know, *this* grant has to do with the English Wikipedia as its focus (that doesn't mean what they find can't be applied to other projects -- quite the opposite).
However, I do recall Erik saying something about a grant that is being worked on for Commons, see http://commons.wikimedia.org/wiki/Commons:Case_for_Commons
Hi,
On Wed, Mar 25, 2009 at 4:12 PM, Casey Brown cbrown1023.ml@gmail.com wrote:
Michael Dale is also doing some pretty amazing job regarding the upload wizard:
http://brianna.modernthings.org/article/184/media-handling-on-wikimedia-prev... http://brianna.modernthings.org/article/180/media-handling-on-wikimedia-prev...
2009/3/25 Casey Brown cbrown1023.ml@gmail.com:
Yep, that's correct. We've submitted a grant proposal specifically with regard to uploading usability (which also involves the complexity of licensing templates and such), and hope to hear back soon. Uploading is not included in these first user tests.
Thanks Erik, and to those who posted links. I'm very glad that Commons is also taken into account with the usability enhancements.
-- Hay
On Wed, Mar 25, 2009 at 5:10 PM, Erik Moeller erik@wikimedia.org wrote:
2009/3/25 John at Darkstar vacuum@jeb.no:
I'm not sure about your definition of "Wikipedians" above, but the recruiting procedure uses a screening process to recruit _readers_ of Wikipedia with no editing experience. Our goal here is to at the end of the day make improvements to convert more readers to editors.
Ten participants are not nearly enough, they can only give you some clue about the real problem.
There are different philosophies of usability, including a philosophy of agile testing with few test subjects ( http://www.useit.com/alertbox/20000319.html ). There's general agreement that with these kinds of tests, you'll quickly see diminishing returns - adding many more people doesn't actually help you discover many more problems. Moreover, resources are not infinite: finding a good balance in terms of the number of testers allows you to conduct more tests later, with the goal of validating whether the changes you've made actually have had the intended effect.
Erik
wikimedia-l@lists.wikimedia.org