"it was difficult for contributors to tell if a survey was ethical, vetted"

Now, that's a problem of bad research design. Survey design 101 requires that an invitation to the survey briefly discusses those issues. It all boils down to the fact that many lazy or inadequately trained researchers don't bother to do what is described at https://en.wikipedia.org/wiki/Wikipedia:Ethically_researching_Wikipedia#Surveys_and_interviews

It is, unfortunately, not in our power to educate those researchers. RCOM cannot do it, because most researchers will never find out it exists, and will send invitations to their surveys or such ignoring any required (or recommended) processes.

There's only one way that a body like RCOM could try to have some real influence among serious Wikipedia researchers who at least have a decent chance to finding out that it exists and what it does (like those of us here). That is, if it had a carrot to go with the stick of (what, exactly, I am still not sure - ban researchers accounts if they don't follow RCOM procedure? Or just frown at them at WikiSym?). The carrot could be a friendly user interface that would give a researcher an easy way to sample population and send surveys to it, in exchange of jumping through the hoops of whatever RCOM procedure creep becomes. People may consider signing up for RCOM review or such if RCOM gives them something of value in return. Until this happens, I don't except RCOM will become more useful or visible than it has been for the past few years; in fact I am predicting the continuation of its decline, as more and more people realize its a toothless and basically unnecessary body.

--
Piotr Konieczny, PhD
http://hanyang.academia.edu/PiotrKonieczny
http://scholar.google.com/citations?user=gdV8_AEAAAAJ
http://en.wikipedia.org/wiki/User:Piotrus
On 7/18/2014 01:55, phoebe ayers wrote:

On Thu, Jul 17, 2014 at 2:38 AM, Heather Ford <hfordsa@gmail.com> wrote:
Agree with Kerry that we really need to have a more flexible process that speaks to the main problem that (I think) RCOM was started to solve i.e. that Wikipedians were getting tired of being continually contacted by researchers to fill out *surveys*.


That's correct, afaik that was the original motivation, along with some of the concerns that Lane/Nathan raised in the other list -- i.e. that it was difficult for contributors to tell if a survey was ethical, vetted, etc. Frankly, I think some long-term contributors just felt jaded -- for a while it seemed there were lots of surveys and studies to try to find out things that seemed intuitively obvious if you were a participant in the community. I think Heather is right that it seems like there have been fewer surveys in recent years, for whatever reason.

Part of the problem is a somewhat subtle demographic one: while contributors to Wikipedia do turn over, so newer contributors will not necessarily have seen lots of surveys, very heavy editors and admins (who are often easier to identify) tend to be long-term participants who might have been surveyed many times. Additionally, the people who follow mailing lists, social media, etc. (or at least the people who speak up on those channels) skew towards very-long-term contributors who have strong opinions and have seen it all before. So, if you advertise your survey on the mailing list, that's the population you get, and that's the feedback you get. (But it's a catch-22; there's not really other obvious mass channels).

Anyway, this is a hard problem without super-obvious solutions, and not one that there's a lot of models for -- very few online projects are simultaneously as open with their data and as interesting for research purposes!

best,
Phoebe


--
* I use this address for lists; send personal messages to phoebe.ayers <at> gmail.com * 


_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l