thanks everybody, good food for thought. Here are my 2 (or 3) cents

==Community research==
I am under the impression that we are currently applying a double standard to community research vs academic research when it comes to SR requests. To name a case that was mentioned in this thread, I am very fond of the work Sarah is doing on gender gap but it's unfortunate that the project wasn't properly documented and reviewed on R:I. As WSC noted earlier on the list, there are issues of privacy and data retention (on top of recruitment methods) that community members are often unaware of. There was a recent case of a community-driven survey (one I was partly involved with as a WMF contact) that failed to display appropriate terms of participation on the landing page. As a result the data collected could not be shared with the rest of the community, which generated some hostile reaction by other individuals who were interested in crunching the data. We should be pointing community members who wish to run research projects involving SR to RCom in the same way as we capture any attempts by external researchers to contact editors without RCom support/approval.

==The ruthless researcher==
A vocal number of community members seem to move from the assumption that research by community members is by definition healthy, useful and unproblematic and external research is by definition intrusive and potentially irrelevant to the real problems we should be focusing on. As a result the RCom's role is often perceived as that of a gatekeeper to protect the community against the ruthless academic researcher. I'd like to think that part of the role of RCom is to change that perception and to help push the idea that if anything we should not stop researchers who want to help understand our communities and crunch our data but support them. Many WP researchers (especially those who study community dynamics) are active Wikimedians (I am one of them, since 2004) and are at least as vital to our community as MediaWiki hackers are. We can decide to introduce measures that raise the barrier for researchers to study our communities, but let's consider that what we know about Wikipedia's editor dynamics/motivation/participation comes almost entirely from scholarly research. What we should start exploring is the idea of actively pitching research questions to researchers, I'd love to see some of the RCom members who are more closely involved with the community take the lead on this initiative. Someone I met at WikiSym was also very excited at this idea and maybe we can start a dedicated taskforce.

==Surveys, surveys, surveys==
I am not a big fan of surveys, I think there is an annoying imbalance in the SR requests we have been reviewing so far: with only a few exceptions most of these requests are for survey participants. I want to make sure that when we say that subject recruitment should be better controlled we distinguish between surveys and experimentation. The Berkman study itself includes a survey but is mainly a behavioral experiment. As I said in earlier discussions I believe the omnibus survey is a theoretical but practically unmanageable solution (I can give you some background of what it takes to run the editor/editor surveys in terms of logistics, translations etc if you're interested). If we only support requests for inclusion in an omnibus survey that will imply the de facto end of surveys by external researchers, but I want to stress that if we entirely shut down SR we are also putting an end to experimentation.

==Recruitment vs recruitment methods==
As others already noted,  the problem with Berkman was the recruitment method, not recruitment per se. It's unfortunate that the method (selective displaying CN banners to eligible users instead of posting user talk messages), timing (after the end of the fundraiser) and design (displaying logos of the research institutions involved) that were designed to minimize disruption and increase the transparency of the campaign were considered unacceptable: we obviously need to take community concerns seriously and rethink how we communicate and gauge community consensus. At the same time I keep hearing that research requests are disruptive, but I haven't seen any direct evidence other than people occasionally reporting this problem. I mean this as a genuine, not a rhetorical question as I am trying to figure out how someone saying "hey we are interested in what you are doing as a Wikipedian" is more disruptive than templated messages telling new users that they did something wrong. I'd like to better understand which classes of users are under particular stress with SR requests and whether, depending on what the target population of a study is, we can adjust  the recruitment method (for example is some method acceptable for newbies but disruptive for admins or highly active editors and vice versa)

Dario

On Dec 14, 2011, at 8:48 AM, Aaron Halfaker wrote:

Mayo, you bring up a very good point.  I too feel that it is the minority of editors who are at all upset with research recruitment on Wikipedia.  Asking about research recruitment on the next editor survey seems like a good idea. 

I might also offer that the real problem here was the method of recruitment, not recruitment itself.  This is something I hope to bring up in our meeting.  I think that the study would have gone much more smoothly if we had a mechanism to request the participation of individual editors that did not appear in such a prominent place like the central notice banner, but instead was more like a personal request to an individual.  

As we consider research recruitment, I want to make sure that the conversation is not framed around the supposition that recruitment itself is the problem.  Instead, I think we are looking at a problem related to the method of recruitment and ensuring that method bother editors as little as possible.  

-Aaron  

On Wed, Dec 14, 2011 at 10:24 AM, Fuster, Mayo <Mayo.Fuster@eui.eu> wrote:
Hello!

I hope you are fine.

Dario I already moved in order that Goran has access to the survey.

WSC your comments and suggestions seems to strongly assume that there is a consensus on the need to "limiting the amount of surveying that Wikimedians are subjected to". Which is the base for this statement?. Do we have any strong indicator to stand that there is too much request or that this is not the case?. At least on the base of Berkman episode, I would not arrive to that conclusion. Certainly, it does not represent my interpretation: I don't think in the community there is a predominance of a rejection attitude. To me developing research is a way to contribute and beneficial to Wikipedia - but again, beyond each impression on community position and each own personal position on this. we don't have a strong indicator or elaborate analysis of the approach of the community toward research.

In something I think we need to reflect on is that in this stage of things - and from Berkman and Sarah experience- researchers can extract the conclusion that it is better to not get in contact with Rcom and it is better not to consult the community on your recruitment method -  you would save much more time and effort . There is something that it is not working, if this is the case. In this regard, I would not think in terms of how to control and limit the amount of research developed (also because it would be very very difficult) but instead value and incentive that it is done in a way in concordance with how Wikipedians view about how should be done (in terms of recruitment process, in terms of open data, in terms of assuring the results arrive to the community, in terms of addressing questions relevant for wikimedia goals, etc) and that is design in a way that could be as much beneficial for the community as possible.

Cheers! Mayo

«·´`·.(*·.¸(`·.¸ ¸.·´)¸.·*).·´`·»
«·´¨*·¸¸« Mayo Fuster Morell ».¸.·*¨`·»
«·´`·.(¸.·´(¸.·* *·.¸)`·.¸).·´`·»

Research Digital Commons Governance: http://www.onlinecreation.info

Fellow Berkman center for Internet and Society. Harvard University.
Postdoctoral Researcher. Institute of Govern and Public Policies. Autonomous University of Barcelona.
Visiting scholar. Internet Interdisciplinary Institute. Open University of Catalonia (UOC).
Member Research Committee. Wikimedia Foundation
Ph.D European University Institute
Visiting researcher (2008). School of information. University of California, Berkeley.

E-mail: mayo.fuster@eui.eu
E-mail: mayofm@cyber.law.harvard.edu
Twitter/Identica: Lilaroja
Skype: mayoneti
Phone United States: 001 - 8576548231
Phone Spanish State: 0034-648877748

Berkman Center
23 Everett Street, 2nd Floor
Cambridge, MA 02138
+1 (617) 495-7547 (Phone)
+1 (617) 495-7641 (Fax)

Personal Postal Address USA:
The Acetarium http://www.acetarium.com/
265 Elm Street - 4
Somerville, MA, USA
02144
________________________________________
From: rcom-l-bounces@lists.wikimedia.org [rcom-l-bounces@lists.wikimedia.org] On Behalf Of WereSpielChequers [werespielchequers@gmail.com]
Sent: 14 December 2011 16:09
To: The Wikimedia Foundation Research Committee mailing list
Subject: Re: [RCom-l] The tragedy of the Commons

Hi Yaroslav,

While I didn't see the actual survey I'm aware that it was run. I suspect that the community would have little problem differentiating between a Wikimedian surveying a targetted group of Wikimedians on currently contentious matters internal to the community as opposed to an outside researcher surveying a large proportion of the community and perhaps asking questions that don't seem very relevant. Sarah's survey could have been done as part of an Omnibus, and I'm sure if we had an Omnibus survey it would be an opportunity to do a followup.

Alternatively we could see it as part of my alternative option of targeted research - unlike the Berkman survey Sarah did her targetting in such a way that she wasn't blocked as spam.....

WSC

On 14 December 2011 14:58, Yaroslav M. Blanter <putevod@mccme.ru<mailto:putevod@mccme.ru>> wrote:
On Wed, 14 Dec 2011 13:46:48 +0000, WereSpielChequers
<werespielchequers@gmail.com<mailto:werespielchequers@gmail.com>> wrote:
> The controversy over Berkman is not in my view primarily a communication
> issue and it certainly isn't about the legitimacy of that survey. I
believe
> that the community trusts RCom as a regulator of research to know
whether
> research is legitimate or not.
>
> A big part of the controversy is over advertising, and I'm not convinced
> that you can design a banner ad for a third party research survey that
> isn't seen by some as advertising for that third party. An Omnibus
survey
> could be a Wikimedia one and therefore I would argue an internal ad
rather
> than a third party one. Perhaps that isn't our only option, and maybe
there
> are alternative ways to solve that, one way would be to change policy to
> allow advertising for bona fide research. But that would be a difficult
one
> to sell to the community, particularly on the heels of a fundraising
drive
> where "Wikipedia doesn't take ads" was a core message.
>
> The other aspect of being a regulator of research is the issue of how we
> control the amount of research requests made to the community. To my
mind
> that is fundamental to what we should be doing, and it is a major reason
> for my being on this committee.  But this is almost an opposite thought
> process to "promoting research".
>
> There are two proposals that I've made as to how we do this, one would
be
> to contact everyone once a year with an Omnibus survey, the other rather
> more complex one is to throttle back research surveying by volume and
limit
> each campaign to a small subset of the community. The two approaches can
be
> hybridised by rewarding institutions that collaborate by allowing them
to
> use our systems to approach a larger proportion of editors. One reason
why
> I was opposed to the Berkman survey was that it was the worst of both
> worlds - one single research project going to all or almost all of our
most
> surveyed community.
>
> I'm not convinced that the community currently has confidence in RCom to
> regulate the amount of research requests that wikimedians and especially
> English language Wikipedians are exposed to. Nor am I convinced that
> everyone on this committee regards that as our responsibility. To my
mind
> this gives us a couple of possibilities, one would be to try and agree a
> mechanism for limiting the amount of surveying that Wikimedians are
> subjected to, and then sell that to the community via a request for
> comment. One option in any such request for comment could be for the
> community to agree not to put any constraints on researchers, but I'd be
> surprised if that option got consensus however strongly it was promoted
by
> some members of RCom. The other possibility would be to clarify that the
> remit of this committee is to promote legitimate research by vetting
> proposals and otherwise communicating with the community; and to inform
the
> community that if it wants to put constraints on legitimate researchers
> contacting wikimedians via the site then it needs a an additional
process
> other than RCom.
>
> WereSpielChequers
>

Thanks for your ideas, which I find very much reasonable. I have an
immediate objection though. Not all research goes through RCom, and we have
no means to stop any single person or organization from sending a hundred
messages to talk pages. For instance, recently it was a survey with the
purpose of understanding the role of the female editors, or whatever the
purpose was (It is difficult for me to find a link immediately, but it can
be done, I guess it was run by Sarah Stierch and colleagues). They did not
bother to go to RCom, and I could imagine what the response were if we
demanded that for instance this survey would become part of Omnibus. Since
it looks almost inevitable that we have to go and ask the community
opinions at some stage, we probably also need to ask this question: Should
every research requiring subject recruitment be regulated (reviewed) by
RCom in advance, or may be the community (first robably of en.wp) just does
not want any regulation of the subject recruitment.

Cheers
Yaroslav

_______________________________________________
RCom-l mailing list
RCom-l@lists.wikimedia.org<mailto:RCom-l@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/rcom-l



The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination, distribution, forwarding, or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited without the express permission of the sender. If you received this communication in error, please contact the sender and delete the material from any computer.


_______________________________________________
RCom-l mailing list
RCom-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/rcom-l

_______________________________________________
RCom-l mailing list
RCom-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/rcom-l