Hi everyone,
Thanks for your thoughtful comments about using survey tools for feature development.
Personally, I find surveys invaluable for learning what users think of the features we develop. Behavioral metrics are wonderful for learning how our users use our tools, but they don’t tell us how they feel about them. Surveys help us get that information quickly from a lot of users, providing useful qualitative feedback to complement our quantitative data.
I have used online surveys successfully throughout my career, and look forward to continuing to use this tool here at Wikimedia, as another important data point to inform our product decisions. They are particularly useful for hearing from our ‘silent majority’ of users — instead of the ‘vocal minority’ that dominates our talk page discussions.
For example, we are using surveys in multiple languages now to get advance feedback about Media Viewer in multiple languages — and the early responses have helped us identify key issues, from the perspective of readers (our largest user group) — not just editors. You can review these first results here - more to come: https://www.mediawiki.org/wiki/Multimedia/Media_Viewer/Survey
To address Mark’s original question as to whether of not to use a survey for Upload Wizard, I believe that surveys are needed for that tool as well, as many media contributors do not edit frequently, if at all, even if they are registered users. For example, research has pointed out that only a small fraction of contributors to Wiki Loves Monuments become editors, and it’s likely that many mobile uploaders may not be editors themselves. Casual contributors are different from editors in many ways, and are also likely to be uncomfortable with talk page tools.
So from my viewpoint, a simple survey popup form seems the most practical way for us to collect that feedback. We want to capture user feedback about Upload Wizard as soon as they have completed their upload, while it’s still fresh on their minds. So I would recommend a prominent invitation to leave feedback on the final step of the upload process, with the popup form opening right over that final page, rather than going to another talk page.
I am open to which survey platform should be used to serve our needs, as long as the platform works reliably, doesn’t require additional development and provides all the standard features I am used to for easily creating surveys, collecting the data, then analyzing, visualizing and sharing the results. Personally, I find that Survey Monkey provides an excellent toolkit for all these functions, which lets me do my work efficiently.
But if another open source tool exists that provides the same services with the same level of quality, I would be happy to consider it. And I have been recommending to Erik and Howie regularly that we consider developing (or adapting) a survey tool that can better integrate with our wikis. However, this is not a trivial task, if we want to match the level of functionality available from other solutions.
It certainly would be worth talking to the folks at LimeSurvey to discuss improvements to their platform, which could probably be adapted for our purposes, perhaps even on a contract basis. I would be happy to participate in this discussion, to help identify key requirements, as a regular survey customer. I should also point out that I love Dario’s 'micro-survey’ idea, a tool which I would use in a minute if it were available — so this might be a direction we might want to explore as part of this investigation.
For now, Survey Monkey offers us a practical and reliable toolkit, which I am comfortable using until we find a better solution. And to answer Tilman’s question, the reason we created separate surveys in multiple languages for the current Media Viewer campaign was so that we could easily track, visualize and share responses for each language independently. Survey Monkey does provide multiple language support (under ’Survey Options’ in the design view), though you still need to have the questions and answers translated separately.
I hope these observations are helpful. I have added a few comments below as well. Please let me know if you would like any more clarifications on why I think surveys are important to our work, and should continue to be used to learn from our users.
Thanks,
Fabrice
On Apr 30, 2014, at 12:47 PM, Andrew Gray andrew.gray@dunelm.org.uk wrote:
On 30 April 2014 08:52, Federico Leva (Nemo) nemowiki@gmail.com wrote:
We do have such a feature in MediaWiki, though: mediawiki.feedback.js. It's just a JavaScrip popup which saves the comment to a page on the wiki.
many of which would not otherwise comment at all. And for some tools (such as UploadWizard) there is no obvious place to leave comments, and opening Bugzilla is a new-tab + multi-step process away.
UploadWizard (like VisualEditor) uses what above. Maybe it needs an option to be offered more prominently under some conditions? This is probably the most viable option here, almost no technical effort and more value in output.
It's certainly simpler to implement, but anything that involves on-wiki recording has two main problems:
- friction in saving the entry (eg edit conflicts, login required,
user IP blocked)
- privacy problems (comments are public and effectively attributed)
This isn't much of an issue for things like "please give us feedback on the new fancy upload tool" - where everyone can be expected to have a functioning account and aware of how the wiki works, but if you're going to be gathering feedback on reader-focused things it breaks down.
Thanks, Andrew, for these thoughtful observations.
I agree with your views.
--
- Andrew Gray
andrew.gray@dunelm.org.uk
On Apr 30, 2014, at 12:52 AM, Federico Leva (Nemo) nemowiki@gmail.com wrote:
Samuel Klein, 30/04/2014 05:35:
Asking 1/1000 users of tool X a single open-ended question ("please give us feedback on X" or "how is X working for you"?) can be a handy way to encourage brief input from a cross-section of users,
We do have such a feature in MediaWiki, though: mediawiki.feedback.js. It's just a JavaScrip popup which saves the comment to a page on the wiki.
This doesn’t seem very practical, for the reasons Andrew outlines above.
many of which would not otherwise comment at all. And for some tools (such as UploadWizard) there is no obvious place to leave comments, and opening Bugzilla is a new-tab + multi-step process away.
UploadWizard (like VisualEditor) uses what above. Maybe it needs an option to be offered more prominently under some conditions? This is probably the most viable option here, almost no technical effort and more value in output.
Tilman Bayer, 29/04/2014 21:58:
might be worth revisiting LimeSurvey, which appears to have undergone a complete rewrite since that installation was removed from WMF servers for security concerns around 2011.
+1. It will need to be done anyway, at some point, e.g. if a general editor survey is tried again.
I support the idea of talking to the LimeSurvey developers, to investigate this further.
But I suspect that some development may be required to meet our requirements, as outlined above.
multilingual support than other solutions [...] lack of integrated language support in Surveymonkey, or just because the focus was on per-project results anyway?
Agreed on all the rest but this point specifically. It seems surveymonkey is really out of question. However, how many languages does Qualtrics support?
Why do you say that Survey Monkey is out of the question? We have used it successfully for other projects before.
LimeSurvey says 50; it is translated on a public instance of GlotPress. GlotPress is from Automattic and is used to make some Wordpress locale, hence some translatewiki.net have experience with it. However I wasn't able to gather much information about it, I only know that it's yet another web tool for .po format; maybe Stu can put us in contact with someone with more insight (especially on how much it's used and how prioritary for Automattic)?
Keep in mind that language support typically means that the standard buttons and error messages are translated in a variety of languages by most platform providers.
All survey contents have to be translated separately. For our purposes, we have successfully used onwiki translation tools to get our surveys translated in other languages.
Nemo
On Apr 29, 2014, at 8:35 PM, Samuel Klein meta.sj@gmail.com wrote:
I think very very simple surveys -- or a standard brightly colored "Feedback" button that's visible from a tool's page -- are useful.
Asking 1/1000 users of tool X a single open-ended question ("please give us feedback on X" or "how is X working for you"?) can be a handy way to encourage brief input from a cross-section of users, many of which would not otherwise comment at all. And for some tools (such as UploadWizard) there is no obvious place to leave comments, and opening Bugzilla is a new-tab + multi-step process away.
Thanks, SJ.
You make some very good points, which I agree with.
SJ
On Tue, Apr 29, 2014 at 3:47 PM, Mark Holmquist mtraceur@member.fsf.org wrote:
On Tue, Apr 29, 2014 at 12:30:57PM -0700, Oliver Keyes wrote:
Geneally speaking my advice to the multimedia team would be "don't go near surveys". I've done a lot of them in the last 3 years, and the one thing I've learned is that surveys are very, very difficult to get right. Another thing I've learned is that if you don't get them right, the results are meaningless and it's hard to tell when that happens.
As I understand it, Jared's team is hiring a qualitatively-focused UX researcher or two in the upcoming budget to do research around design and feature usage; we should hold off until they come in, first because they're simply going to be better at it than we are, and second because it's probably going to be frustrating for them if they come in and find a tool locked in as How We Do Things (and frustrating for us if they want to change that tool):
Hi, Multimedia list!
Just thought I'd cross-post this reply to my call for help with surveys [0] on the analytics list.
I'm sort of of the mind that skipping the survey for UploadWizard is a good idea, especially now that I've thought about it more - using a survey from a third-party site is silly for logged-in users, because logged-in users will know how to use the talk pages and/or bugzilla.
Thoughts?
[0] http://lists.wikimedia.org/pipermail/analytics/2014-April/001911.html
-- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
-- Samuel Klein @metasj w:user:sj +1 617 529 4266
Multimedia mailing list Multimedia@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/multimedia
_______________________________
Fabrice Florin Product Manager Wikimedia Foundation