The first battery of user tests have completed against the Winter prototype. I have included links to all of them at the main Winter page:
https://www.mediawiki.org/wiki/Winter#Harness_One:_Winter
All tests are annotated, but highlights are included in-line.
The primary take-aways are:
* No tester had difficulty recognizing the search bar. * All testers quickly grasped the in-page context action ribbon. * Several testers expressed surprise that Wikipedia had discussions, which lends even more credence to the idea that Vector tabs are effectively invisible. * The "Cancel" button from the editor may need work. * All testers found their way to the contributions page, even when faced with bugs/difficulties. They just did it in other routes.
I am very pleased with the results.
I’ve created a Flow discussion about this here:
https://www.mediawiki.org/w/index.php?title=Talk:Winter&topic_postId=rqj...
--- Brandon Harris, Senior Designer, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
lol @ my, erm, heterogeneous list of contributions - Ukrainian riot police and basketball uniforms! :)
Watching these made me think about one thing we should probably start to pay more attention to: the network of usertesting.com users we've built up over the years. I've definitely seen two of these folks (misterbeauds and kayaker) before in usertesting tests I've run, which means they've had an unusual amount of exposure to the details of Wikipedia UI, both current and planned. That probably partly explains why they seem to know about things like page history, watchlist, etc., without being "real" editors.
Every time I put a test up on usertesting.com, I get at least one of those repeat testers - probably because our Wikipedia tests are a lot more fun than whatever standard eyeball monetization software these people normally have to endure ;) But that means we might be underestimating their level of expertise (since they're not truly naive users) and over-representing their views of our UI improvements (since they tend to offer up similar soundbites time after time - e.g., kayaker's "this looks like code" and "I don't want to mess anything up," which I've heard her say before).
Anyway, just something for our future Lead UX Researcher to tackle, probably with a detailed database/registry of past testers :)
On Fri, Mar 7, 2014 at 7:01 PM, Brandon Harris bharris@wikimedia.orgwrote:
The first battery of user tests have completed against the Winterprototype. I have included links to all of them at the main Winter page:
https://www.mediawiki.org/wiki/Winter#Harness_One:_Winter All tests are annotated, but highlights are included in-line. The primary take-aways are: * No tester had difficulty recognizing the search bar. * All testers quickly grasped the in-page context actionribbon. * Several testers expressed surprise that Wikipedia had discussions, which lends even more credence to the idea that Vector tabs are effectively invisible. * The "Cancel" button from the editor may need work. * All testers found their way to the contributions page, even when faced with bugs/difficulties. They just did it in other routes.
I am very pleased with the results. I've created a Flow discussion about this here:https://www.mediawiki.org/w/index.php?title=Talk:Winter&topic_postId=rqj...
Brandon Harris, Senior Designer, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
Wmfproduct mailing list Wmfproduct@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfproduct
Good things to be aware of, thanks Maryana
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation
M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Mon, Mar 10, 2014 at 2:20 PM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
lol @ my, erm, heterogeneous list of contributions - Ukrainian riot police and basketball uniforms! :)
Watching these made me think about one thing we should probably start to pay more attention to: the network of usertesting.com users we've built up over the years. I've definitely seen two of these folks (misterbeauds and kayaker) before in usertesting tests I've run, which means they've had an unusual amount of exposure to the details of Wikipedia UI, both current and planned. That probably partly explains why they seem to know about things like page history, watchlist, etc., without being "real" editors.
Every time I put a test up on usertesting.com, I get at least one of those repeat testers - probably because our Wikipedia tests are a lot more fun than whatever standard eyeball monetization software these people normally have to endure ;) But that means we might be underestimating their level of expertise (since they're not truly naive users) and over-representing their views of our UI improvements (since they tend to offer up similar soundbites time after time - e.g., kayaker's "this looks like code" and "I don't want to mess anything up," which I've heard her say before).
Anyway, just something for our future Lead UX Researcher to tackle, probably with a detailed database/registry of past testers :)
On Fri, Mar 7, 2014 at 7:01 PM, Brandon Harris bharris@wikimedia.orgwrote:
The first battery of user tests have completed against the Winterprototype. I have included links to all of them at the main Winter page:
https://www.mediawiki.org/wiki/Winter#Harness_One:_Winter All tests are annotated, but highlights are included in-line. The primary take-aways are: * No tester had difficulty recognizing the search bar. * All testers quickly grasped the in-page context actionribbon. * Several testers expressed surprise that Wikipedia had discussions, which lends even more credence to the idea that Vector tabs are effectively invisible. * The "Cancel" button from the editor may need work. * All testers found their way to the contributions page, even when faced with bugs/difficulties. They just did it in other routes.
I am very pleased with the results. I've created a Flow discussion about this here:https://www.mediawiki.org/w/index.php?title=Talk:Winter&topic_postId=rqj...
Brandon Harris, Senior Designer, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
Wmfproduct mailing list Wmfproduct@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfproduct
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
I've done most of the leg work now for turning this into a beta feature. I made a minor but super cool and exciting change to it that will blow your mind - watch this space :)
Reviews welcomed here - https://gerrit.wikimedia.org/r/107523 !
On Mon, Mar 10, 2014 at 2:55 PM, Jared Zimmerman < jared.zimmerman@wikimedia.org> wrote:
Good things to be aware of, thanks Maryana
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Mon, Mar 10, 2014 at 2:20 PM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
lol @ my, erm, heterogeneous list of contributions - Ukrainian riot police and basketball uniforms! :)
Watching these made me think about one thing we should probably start to pay more attention to: the network of usertesting.com users we've built up over the years. I've definitely seen two of these folks (misterbeauds and kayaker) before in usertesting tests I've run, which means they've had an unusual amount of exposure to the details of Wikipedia UI, both current and planned. That probably partly explains why they seem to know about things like page history, watchlist, etc., without being "real" editors.
Every time I put a test up on usertesting.com, I get at least one of those repeat testers - probably because our Wikipedia tests are a lot more fun than whatever standard eyeball monetization software these people normally have to endure ;) But that means we might be underestimating their level of expertise (since they're not truly naive users) and over-representing their views of our UI improvements (since they tend to offer up similar soundbites time after time - e.g., kayaker's "this looks like code" and "I don't want to mess anything up," which I've heard her say before).
Anyway, just something for our future Lead UX Researcher to tackle, probably with a detailed database/registry of past testers :)
On Fri, Mar 7, 2014 at 7:01 PM, Brandon Harris bharris@wikimedia.orgwrote:
The first battery of user tests have completed against theWinter prototype. I have included links to all of them at the main Winter page:
https://www.mediawiki.org/wiki/Winter#Harness_One:_Winter
All tests are annotated, but highlights are included in-line. The primary take-aways are: * No tester had difficulty recognizing the search bar. * All testers quickly grasped the in-page context actionribbon. * Several testers expressed surprise that Wikipedia had discussions, which lends even more credence to the idea that Vector tabs are effectively invisible. * The "Cancel" button from the editor may need work. * All testers found their way to the contributions page, even when faced with bugs/difficulties. They just did it in other routes.
I am very pleased with the results. I've created a Flow discussion about this here:https://www.mediawiki.org/w/index.php?title=Talk:Winter&topic_postId=rqj...
Brandon Harris, Senior Designer, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
Wmfproduct mailing list Wmfproduct@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfproduct
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
Test server for the non-gerrit inclined?
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation
M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Wed, Mar 12, 2014 at 11:32 AM, Jon Robson jdlrobson@gmail.com wrote:
I've done most of the leg work now for turning this into a beta feature. I made a minor but super cool and exciting change to it that will blow your mind - watch this space :)
Reviews welcomed here - https://gerrit.wikimedia.org/r/107523 !
On Mon, Mar 10, 2014 at 2:55 PM, Jared Zimmerman < jared.zimmerman@wikimedia.org> wrote:
Good things to be aware of, thanks Maryana
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Mon, Mar 10, 2014 at 2:20 PM, Maryana Pinchuk mpinchuk@wikimedia.orgwrote:
lol @ my, erm, heterogeneous list of contributions - Ukrainian riot police and basketball uniforms! :)
Watching these made me think about one thing we should probably start to pay more attention to: the network of usertesting.com users we've built up over the years. I've definitely seen two of these folks (misterbeauds and kayaker) before in usertesting tests I've run, which means they've had an unusual amount of exposure to the details of Wikipedia UI, both current and planned. That probably partly explains why they seem to know about things like page history, watchlist, etc., without being "real" editors.
Every time I put a test up on usertesting.com, I get at least one of those repeat testers - probably because our Wikipedia tests are a lot more fun than whatever standard eyeball monetization software these people normally have to endure ;) But that means we might be underestimating their level of expertise (since they're not truly naive users) and over-representing their views of our UI improvements (since they tend to offer up similar soundbites time after time - e.g., kayaker's "this looks like code" and "I don't want to mess anything up," which I've heard her say before).
Anyway, just something for our future Lead UX Researcher to tackle, probably with a detailed database/registry of past testers :)
On Fri, Mar 7, 2014 at 7:01 PM, Brandon Harris bharris@wikimedia.orgwrote:
The first battery of user tests have completed against theWinter prototype. I have included links to all of them at the main Winter page:
https://www.mediawiki.org/wiki/Winter#Harness_One:_Winter
All tests are annotated, but highlights are included in-line. The primary take-aways are: * No tester had difficulty recognizing the search bar. * All testers quickly grasped the in-page contextaction ribbon. * Several testers expressed surprise that Wikipedia had discussions, which lends even more credence to the idea that Vector tabs are effectively invisible. * The "Cancel" button from the editor may need work. * All testers found their way to the contributions page, even when faced with bugs/difficulties. They just did it in other routes.
I am very pleased with the results. I've created a Flow discussion about this here:https://www.mediawiki.org/w/index.php?title=Talk:Winter&topic_postId=rqj...
Brandon Harris, Senior Designer, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
Wmfproduct mailing list Wmfproduct@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfproduct
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
-- Jon Robson
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
It needs to get merged first then I'm more than happy to get this setup. On 12 Mar 2014 14:35, "Jared Zimmerman" jared.zimmerman@wikimedia.org wrote:
Test server for the non-gerrit inclined?
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Wed, Mar 12, 2014 at 11:32 AM, Jon Robson jdlrobson@gmail.com wrote:
I've done most of the leg work now for turning this into a beta feature. I made a minor but super cool and exciting change to it that will blow your mind - watch this space :)
Reviews welcomed here - https://gerrit.wikimedia.org/r/107523 !
On Mon, Mar 10, 2014 at 2:55 PM, Jared Zimmerman < jared.zimmerman@wikimedia.org> wrote:
Good things to be aware of, thanks Maryana
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Mon, Mar 10, 2014 at 2:20 PM, Maryana Pinchuk <mpinchuk@wikimedia.org
wrote:
lol @ my, erm, heterogeneous list of contributions - Ukrainian riot police and basketball uniforms! :)
Watching these made me think about one thing we should probably start to pay more attention to: the network of usertesting.com users we've built up over the years. I've definitely seen two of these folks (misterbeauds and kayaker) before in usertesting tests I've run, which means they've had an unusual amount of exposure to the details of Wikipedia UI, both current and planned. That probably partly explains why they seem to know about things like page history, watchlist, etc., without being "real" editors.
Every time I put a test up on usertesting.com, I get at least one of those repeat testers - probably because our Wikipedia tests are a lot more fun than whatever standard eyeball monetization software these people normally have to endure ;) But that means we might be underestimating their level of expertise (since they're not truly naive users) and over-representing their views of our UI improvements (since they tend to offer up similar soundbites time after time - e.g., kayaker's "this looks like code" and "I don't want to mess anything up," which I've heard her say before).
Anyway, just something for our future Lead UX Researcher to tackle, probably with a detailed database/registry of past testers :)
On Fri, Mar 7, 2014 at 7:01 PM, Brandon Harris bharris@wikimedia.orgwrote:
The first battery of user tests have completed against theWinter prototype. I have included links to all of them at the main Winter page:
https://www.mediawiki.org/wiki/Winter#Harness_One:_Winter
All tests are annotated, but highlights are included in-line. The primary take-aways are: * No tester had difficulty recognizing the search bar. * All testers quickly grasped the in-page contextaction ribbon. * Several testers expressed surprise that Wikipedia had discussions, which lends even more credence to the idea that Vector tabs are effectively invisible. * The "Cancel" button from the editor may need work. * All testers found their way to the contributions page, even when faced with bugs/difficulties. They just did it in other routes.
I am very pleased with the results. I've created a Flow discussion about this here:https://www.mediawiki.org/w/index.php?title=Talk:Winter&topic_postId=rqj...
Brandon Harris, Senior Designer, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
Wmfproduct mailing list Wmfproduct@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wmfproduct
-- Maryana Pinchuk Product Manager, Wikimedia Foundation wikimediafoundation.org
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
-- Jon Robson
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
For the final round of preliminary testing on Winter before me move on to a full beta feature I think we need to make a few changes...
Winter next steps testing:
- make sure in-page icons match upper right nav icons, not the user actions (outlined vs filled in icons) - have mouse over hover labels for upper right tools - Need clearer "back to article action" from discussion, history, maybe try some rough breadcrumb like thing just to test? - remove the winter version number in the left nav - remove the save my name from login - can user and contribs page use the user name typed by the user? - change "login screen" to say:
/// Current
This is an interactive prototype for *Winter*, a beta feature that modifies the default skin for Wikipedia.
Please read a document about this prototypehttp://www.mediawiki.org/wiki/Winter .
*Nothing you enter here will be saved.* This is a pure-Javascript tool and everything will reset if you reload the page.
Before we begin, we'll need to know what to call you.
What is your user name?
/// Proposed
This is an interactive prototype being used to test updates to the interface for Wikipedia. It is in progress and you may run into bugs or issue. Remember we're testing the system, not you, so have fun and try to act as you normally would on the site.
This is only a prototype, and nothing you do here will harm Wikipedia, or be saved by the system.
*Please choose a user name to personalize your experience using the prototype:*
As soon as we can make these changes I'd like to launch the 3rd and final set of small scale user tests before taking our learnings from these and making some larger changes and releasing the beta feature.
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation
M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Fri, Mar 7, 2014 at 7:01 PM, Brandon Harris bharris@wikimedia.orgwrote:
The first battery of user tests have completed against the Winterprototype. I have included links to all of them at the main Winter page:
https://www.mediawiki.org/wiki/Winter#Harness_One:_Winter All tests are annotated, but highlights are included in-line. The primary take-aways are: * No tester had difficulty recognizing the search bar. * All testers quickly grasped the in-page context actionribbon. * Several testers expressed surprise that Wikipedia had discussions, which lends even more credence to the idea that Vector tabs are effectively invisible. * The "Cancel" button from the editor may need work. * All testers found their way to the contributions page, even when faced with bugs/difficulties. They just did it in other routes.
I am very pleased with the results. I've created a Flow discussion about this here:https://www.mediawiki.org/w/index.php?title=Talk:Winter&topic_postId=rqj...
Brandon Harris, Senior Designer, Wikimedia Foundation
Support Free Knowledge: http://wikimediafoundation.org/wiki/Donate
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design
On Thu, Mar 13, 2014 at 1:40 PM, Jared Zimmerman < jared.zimmerman@wikimedia.org> wrote:
For the final round of preliminary testing on Winter before me move on to a full beta feature I think we need to make a few changes…
How are we doing to incorporate the mobile changes that Jon has made into the beta experiment? I want to make sure were set for success on day 1 here.
--tomasz
On Thu, Mar 13, 2014 at 2:16 PM, Tomasz Finc tfinc@wikimedia.org wrote:
How are we doing to incorporate the mobile changes that Jon has made into the beta experiment? I want to make sure were set for success on day 1 here.
It's awaiting review in gerrit and should go out whenever the actual beta feature does. I don't think it's yet a part of the usability tests.
Tomasz, no its not part of the usability tests, we'll make lots of iterative changes while its in Beta Features, these 3 rounds of usability testing are just to make sure we start in a good place and haven't made and grave missteps that put us in a bad place to start from. e.g. people can't find search, personal bar, or talk/history.
*Jared Zimmerman * \ Director of User Experience \ Wikimedia Foundation
M : +1 415 609 4043 | : @JaredZimmermanhttps://twitter.com/JaredZimmerman
On Thu, Mar 13, 2014 at 2:20 PM, Steven Walling swalling@wikimedia.orgwrote:
On Thu, Mar 13, 2014 at 2:16 PM, Tomasz Finc tfinc@wikimedia.org wrote:
How are we doing to incorporate the mobile changes that Jon has made into the beta experiment? I want to make sure were set for success on day 1 here.
It's awaiting review in gerrit and should go out whenever the actual beta feature does. I don't think it's yet a part of the usability tests.
-- Steven Walling, Product Manager https://wikimediafoundation.org/
Design mailing list Design@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/design