Hi,
Once upon a time, the Multimedia team set out on a great quest [0] to add a survey to the Media Viewer they were building. This quest ended with the team deciding that the great Lord SurveyMonkey would provide the users the survey they needed with the least amount of effort, and there was much rejoicing.
But a dark threat loomed over the land. With one product using SurveyMonkey, other products seemed poised to use it, too [1]. The compromise built upon the premise that Media Viewer needed a survey in less time than it would take to find and set up a free [2] solution was beginning to bleed over into other projects where no such time crunch was present.
Our heroes now reach out to their friends in other realms [3]. Is there hope for freedom in the land of getting user feedback? Will MediaWiki or the grander Wikimedia ecosystem soon have a survey tool that all projects can use with minimal hassle?
I'd appreciate looking at existing solutions and planned non-existing ones, and I offer some of my ample [4] free time to help with setting something up.
[0] https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/261 [1] https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/522 [2] https://gnu.org/philosophy/free-sw.html [3] http://mediawiki.org/wiki/Analytics [4] https://en.wikipedia.org/wiki/Sarcasm
Cheers,
On 04/29/2014 01:34 PM, Mark Holmquist wrote:
Our heroes now reach out to their friends in other realms [3]. Is there hope for freedom in the land of getting user feedback? Will MediaWiki or the grander Wikimedia ecosystem soon have a survey tool that all projects can use with minimal hassle?
The only survey I've done was one for gender, on signup. It was designed to see the impact of VisualEditor. That used GuidedTour as a modal library (not really its original purpose), and just had the buttons fire EventLogging requests.
Basically, this idea could work with any modal library. Crunching the data is pretty easy if there are just two buttons, but it would get more complicated if more questions were asked. And this doesn't address free-form text at all.
Matt Flaschen
On Tue, Apr 29, 2014 at 02:08:24PM -0400, Matthew Flaschen wrote:
The only survey I've done was one for gender, on signup. It was designed to see the impact of VisualEditor. That used GuidedTour as a modal library (not really its original purpose), and just had the buttons fire EventLogging requests.
Basically, this idea could work with any modal library. Crunching the data is pretty easy if there are just two buttons, but it would get more complicated if more questions were asked. And this doesn't address free-form text at all.
Yeah, that won't really cut it, we'd prefer to have something a little more full-featured and flexible, plus we'd like to not have to re-write the framework every time we do it. This would ideally be a service with an API client library in core, or similar.
On Tue, Apr 29, 2014 at 10:34 AM, Mark Holmquist mtraceur@member.fsf.orgwrote:
But a dark threat loomed over the land. With one product using SurveyMonkey, other products seemed poised to use it, too [1]. The compromise built upon the premise that Media Viewer needed a survey in less time than it would take to find and set up a free [2] solution was beginning to bleed over into other projects where no such time crunch was present.
Our heroes now reach out to their friends in other realms [3]. Is there hope for freedom in the land of getting user feedback? Will MediaWiki or the grander Wikimedia ecosystem soon have a survey tool that all projects can use with minimal hassle?
This is a longstanding problem. *TL;DR*: there isn't a good existing survey tool that suits our architecture, and we honestly don't care enough about surveys to build one in-house. Possible exceptions include the tools for surveying built during the original Usability Initiative or built in to VisualEditor during the alpha releases.
For other large surveys, like the annual user survey we used to run via CentralNotice banners,[1] we've also used a third party solution, Qualtrics. I think we still pay for it actually. SurveyMonkey is actually a much better system, since we don't necessarily have to pay for it, and it's easier to use for survey takers and creators.
In the past, we experimented with hosting FOSS survey tools like LimeSurvey,[2] but as far as I know, it was disabled because ops identified security problems with it. Maybe this has been fixed upstream? A cursory search of GitHub and StackOverflow doesn't show many other usable, actively-developed options in terms of a PHP-based FOSS survey engine.[3]
In my view, we have much bigger pain points in gathering data, such as lack of a framework for A/B testing. Using SurveyMonkey is the least of our problems in gathering qualitative or quantitative data to make decisions with.
1. https://meta.wikimedia.org/wiki/Category:Surveys 2. https://github.com/LimeSurvey 3. https://stackoverflow.com/questions/3648843/open-source-php-form-survey-engi...
We reviewed building a survey tool a couple of quarterly reviews ago and like Stephen said it wasn't prioritized highly compared to many other requests. This, combined with the availability of sub-optimal but workable solutions like SurveyMonkey makes it unlikely we'll look into building one in the foreseeable future.
That said, "adopting" an existing OSS tool like LimeSurvey would probably be the route we would choose. If the need is great enough we could also investigate working with an OSS project to sponsor desired functionality.
-Toby
On Tue, Apr 29, 2014 at 11:30 AM, Steven Walling swalling@wikimedia.orgwrote:
On Tue, Apr 29, 2014 at 10:34 AM, Mark Holmquist mtraceur@member.fsf.orgwrote:
But a dark threat loomed over the land. With one product using SurveyMonkey, other products seemed poised to use it, too [1]. The compromise built upon the premise that Media Viewer needed a survey in less time than it would take to find and set up a free [2] solution was beginning to bleed over into other projects where no such time crunch was present.
Our heroes now reach out to their friends in other realms [3]. Is there hope for freedom in the land of getting user feedback? Will MediaWiki or the grander Wikimedia ecosystem soon have a survey tool that all projects can use with minimal hassle?
This is a longstanding problem. *TL;DR*: there isn't a good existing survey tool that suits our architecture, and we honestly don't care enough about surveys to build one in-house. Possible exceptions include the tools for surveying built during the original Usability Initiative or built in to VisualEditor during the alpha releases.
For other large surveys, like the annual user survey we used to run via CentralNotice banners,[1] we've also used a third party solution, Qualtrics. I think we still pay for it actually. SurveyMonkey is actually a much better system, since we don't necessarily have to pay for it, and it's easier to use for survey takers and creators.
In the past, we experimented with hosting FOSS survey tools like LimeSurvey,[2] but as far as I know, it was disabled because ops identified security problems with it. Maybe this has been fixed upstream? A cursory search of GitHub and StackOverflow doesn't show many other usable, actively-developed options in terms of a PHP-based FOSS survey engine.[3]
In my view, we have much bigger pain points in gathering data, such as lack of a framework for A/B testing. Using SurveyMonkey is the least of our problems in gathering qualitative or quantitative data to make decisions with.
https://stackoverflow.com/questions/3648843/open-source-php-form-survey-engi...
-- Steven Walling, Product Manager https://wikimediafoundation.org/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
In my view, we have much bigger pain points in gathering data, such as lack of a framework for A/B testing. Using SurveyMonkey is the least of our problems in gathering qualitative or quantitative data to make decisions with.
Agreed, bigger fish to fry unfortunately, though I agree there's a need for a survey system.
On 04/29/2014 11:30 AM, Steven Walling wrote:
On Tue, Apr 29, 2014 at 10:34 AM, Mark Holmquist <mtraceur@member.fsf.org mailto:mtraceur@member.fsf.org> wrote:
But a dark threat loomed over the land. With one product using SurveyMonkey, other products seemed poised to use it, too [1]. The compromise built upon the premise that Media Viewer needed a survey in less time than it would take to find and set up a free [2] solution was beginning to bleed over into other projects where no such time crunch was present. Our heroes now reach out to their friends in other realms [3]. Is there hope for freedom in the land of getting user feedback? Will MediaWiki or the grander Wikimedia ecosystem soon have a survey tool that all projects can use with minimal hassle?
For other large surveys, like the annual user survey we used to run via CentralNotice banners,[1] we've also used a third party solution, Qualtrics. I think we still pay for it actually. SurveyMonkey is actually a much better system, since we don't necessarily have to pay for it, and it's easier to use for survey takers and creators.
Mark, If you haven't looked at Qualtrics in the past, check it out. It has many more options than SurveyMonkey that can make it worthwhile specially if we are already paying for it.
Leila
https://stackoverflow.com/questions/3648843/open-source-php-form-survey-engi...
-- Steven Walling, Product Manager https://wikimediafoundation.org/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Tue, Apr 29, 2014 at 11:56:58AM -0700, Leila Zia wrote:
Mark, If you haven't looked at Qualtrics in the past, check it out. It has many more options than SurveyMonkey that can make it worthwhile specially if we are already paying for it.
On Tue, Apr 29, 2014 at 10:34:21AM -0700, Mark Holmquist wrote:
free [2] [2] https://gnu.org/philosophy/free-sw.html
Qualtrics is no better, AFAICT from a cursory glance at their Wikipedia page and website.
From a cursory glance at their website, sure. From having actually used
both of them, Qualtrics is far superior in its featureset and how that featureset is presented, although surveymonkey has got a lot better recently.
Geneally speaking my advice to the multimedia team would be "don't go near surveys". I've done a lot of them in the last 3 years, and the one thing I've learned is that surveys are very, very difficult to get right. Another thing I've learned is that if you don't get them right, the results are meaningless and it's hard to tell when that happens.
As I understand it, Jared's team is hiring a qualitatively-focused UX researcher or two in the upcoming budget to do research around design and feature usage; we should hold off until they come in, first because they're simply going to be better at it than we are, and second because it's probably going to be frustrating for them if they come in and find a tool locked in as How We Do Things (and frustrating for us if they want to change that tool):
On 29 April 2014 12:06, Mark Holmquist mtraceur@member.fsf.org wrote:
On Tue, Apr 29, 2014 at 11:56:58AM -0700, Leila Zia wrote:
Mark, If you haven't looked at Qualtrics in the past, check it out. It has many more options than SurveyMonkey that can make it worthwhile specially if we are already paying for it.
On Tue, Apr 29, 2014 at 10:34:21AM -0700, Mark Holmquist wrote:
free [2] [2] https://gnu.org/philosophy/free-sw.html
Qualtrics is no better, AFAICT from a cursory glance at their Wikipedia page and website.
-- Mark Holmquist Software Engineer, Multimedia Wikimedia Foundation mtraceur@member.fsf.org https://wikimediafoundation.org/wiki/User:MHolmquist
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQIcBAEBAgAGBQJTX/g4AAoJEEPl+wghkjzxGnQP/R+rPdcLbm/t7V6StDDHa5zx zDy0WqgQ6JZxQJ8ysuYjV6Dsg9TIohuyt9F7RC01L9LWFqfYf2UXCP9uu6SRNQEV 2Lk5yddrEeAgi7rwChpP5bT2hXQZSZM2RTc55bWzURt3A+haH8ivsEPtUhUnMStj 3NP95qoxl8XyPdIp7sQMCIlxNcOfwPm7cUR3NRd0RhxQZf0jUYhQHPABm5aLY0VA jFBz0b1WmjEMXh4TnMRLeXpdQQmM3qSHFqrLKG1kcqJZrG6vNOMPxauqwCT0d10D 24pTIp5E9AOyk04UWt078QKW1+7W9nkODVvzB+vpsvEeOcVvVAezP4r6VRwXoS08 bOPT6TIy4AMSXBS11I6ToAWsFxODhZLHQOt/93N147zuouHVBfFwCUdawNRogc9a YBN0GKT6J4m9MSIdEuGpt54JLxjbXir2nfX1RF8SRSbBRzYOC8ZS4Deuom8Q675c Vb633n8uDPlU07Oqfn/ge4+HtDr2bQLpo0AaAjCr8XHqsCOdWa/FjwVUW+FKH4Ro hbpmm7NDxGvxT0p0hYuZYP+WfLYZ0G50VOqEZ9S2WZsUrLIvfUH2N9aJmDoZGEpt VRUIi+1sfXaje0fxDUMRzLrZCon2/Sdc2L1pDutQ56tvNslBQm9kLcHzikbF6WgK z2cisbGz5L93yb60tmLm =MsE0 -----END PGP SIGNATURE-----
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Tue, Apr 29, 2014 at 12:30 PM, Oliver Keyes okeyes@wikimedia.org wrote:
Geneally speaking my advice to the multimedia team would be "don't go near surveys". I've done a lot of them in the last 3 years, and the one thing I've learned is that surveys are very, very difficult to get right. Another thing I've learned is that if you don't get them right, the results are meaningless and it's hard to tell when that happens.
Oh, surveys aren't all that hard to get right ;) And I bet your surveys were mostly fine, Oliver. I'm happy to help with survey design if anyone has questions.
- J
Says the guy with a HCI doctorate. Paging doctors Dunning and Krueger :P.
The crux of my argument, though, is that I'm uncomfortable with us saying "yes, let's build/standardise on a tool for qualitative analysis" when we're actively recruiting for several qualitative analysts: it's unfair for us to make decisions for them, unless a survey about MediaViewer really can't wait a couple of months.
On 29 April 2014 13:20, Jonathan Morgan jmorgan@wikimedia.org wrote:
On Tue, Apr 29, 2014 at 12:30 PM, Oliver Keyes okeyes@wikimedia.orgwrote:
Geneally speaking my advice to the multimedia team would be "don't go near surveys". I've done a lot of them in the last 3 years, and the one thing I've learned is that surveys are very, very difficult to get right. Another thing I've learned is that if you don't get them right, the results are meaningless and it's hard to tell when that happens.
Oh, surveys aren't all that hard to get right ;) And I bet your surveys were mostly fine, Oliver. I'm happy to help with survey design if anyone has questions.
- J
-- Jonathan T. Morgan Learning Strategist Wikimedia Foundation User:Jtmorgan https://wikimediafoundation.org/wiki/User:Jtmorgan jmorgan@wikimedia.org
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
On Tue, Apr 29, 2014 at 12:06 PM, Mark Holmquist mtraceur@member.fsf.org wrote:
On Tue, Apr 29, 2014 at 11:56:58AM -0700, Leila Zia wrote:
Mark, If you haven't looked at Qualtrics in the past, check it out. It has many more options than SurveyMonkey that can make it worthwhile specially if we are already paying for it.
On Tue, Apr 29, 2014 at 10:34:21AM -0700, Mark Holmquist wrote:
free [2] [2] https://gnu.org/philosophy/free-sw.html
Qualtrics is no better, AFAICT from a cursory glance at their Wikipedia page and website.
See also: https://meta.wikimedia.org/wiki/Programs_talk:Evaluation_portal/Boiler_room#... https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Shop/Qualtrics
Yes, it's of course a very proprietary solution, unfortunately. And especially for smaller surveys, it might be worth revisiting LimeSurvey, which appears to have undergone a complete rewrite since that installation was removed from WMF servers for security concerns around 2011.
That said, we also pay Qualtrics as a service and platform; any honest comparison with a free software tool will need to figure in the work of installing and running the software ourselves. On the other hand, an in-house installation would have privacy advantages (WMF staff: LCA has an office wiki page with survey requirements; I guess the Multimedia team has been relying on an existing, already approved contract).
As Leila says, Qualtrics offers many reporting options that can make one's life much easier. I should say that with the last editor survey, we encountered quite a few bugs, quirks and other issues with this and other Qualtrics features (some due to the size of the dataset). But that might be even worse with other tools, and Qualtrics also has a responsive tech support.
Another point is that Qualtrics appears to offer better multilingual support than other solutions - they even added extra interface languages for us (WMF) in 2011. SurveyMonkey still does not support RTL languages such as Arabic and Hebrew. Qualtrics does, albeit with some quirks.
It appears that the Mediaviewer survey ( https://www.mediawiki.org/wiki/Multimedia/Media_Viewer/Survey ) was run separately for different languages, instead of as one multilingual survey. Was this due to a lack of integrated language support in Surveymonkey, or just because the focus was on per-project results anyway?
On Tue, Apr 29, 2014 at 10:34 AM, Mark Holmquist mtraceur@member.fsf.orgwrote:
Our heroes now reach out to their friends in other realms [3]. Is there hope for freedom in the land of getting user feedback? Will MediaWiki or the grander Wikimedia ecosystem soon have a survey tool that all projects can use with minimal hassle?
The grantmaking realm would be supportive of such a conquest and an eager partaker of the plunder.
... even a simple polling feature would be useful for us to get quick user feedback, but we definitely have use cases for more in-depth surveys!