A few clarifications on the RCom review procedure (cc'ing Lane as he was raising a similar point here [1])

The current review process for subject recruitment (SR) requests (as well as other types of research requests) has been set up since the creation of the Research Index. It has been adopted since then by the Research Committee and by the Foundation as the de facto standard process through which all such requests are processed – whether they are from external researchers, WMF staff or community members. Despite the lack of a formal policy – which is partly due to the fact that a broadly agreed solution on how to manage SR requests has never been found in years [2] – this process was introduced and applied to all SR proposals as the minimum requirement to:

• help document these requests and the credentials of their authors • ensure that proposals are legitimate, and that recruitment messages are not used for abuse • ensure that they meet basic requirements of privacy, data retention and data licensing • assess whether the proposed recruitment strategy and sampling requirements are sensible • identify, wherever possible, redundant or potentially disruptive research • point the researcher to existing work on the topic • mitigate the "survey burnout" that affects our editor community

It should be stressed that, precisely due to the lack of a formal policy, the RCom has never been in a position to grant any kind of "definitive approval" to recruit participants: the best we can do is to flag a proposal as "reviewed" or help identify and report patently abusive requests. All proposals that are submitted to our attention are automatically marked as "pending review" via a dedicated template (which will display a yellow SR icon and add the proposal to the appropriate category [3]). We strive to provide the above kind of support and assessment to the different requests we receive and once a project is reviewed we change the support flag to "reviewed" . This is not to say that the process is 100% error-free or very efficient (we unfortunately have little bandwidth to dedicate to this process and review all requests in a timely way), but the review itself (if you haven't come across one [4]) tends to be quite serious and exhaustive.

So to briefly answer Pine's questions:

- yes, going through RCom review is the standard procedure we expect all proposals to comply with - no, a proposal should not be removed from Meta if it hasn't been reviewed, it should only be flagged as pending review using the WMF-support template. This also means that Audrey fully complied with the expected procedure to submit a SR request. - yes, there are privacy concerns, and this is the reason why we take the review of data collection/retention/licensing terms in the proposal very seriously. As these surveys do not fall under the WMF's privacy policy they are not reviewed by WMF Legal team unless they are considered potentially abusive. The only privacy terms that apply are those displayed on the landing page of a survey and our goal is to support best practices in setting up these terms (for example, by making sure that these terms are explicitly accepted by the participants before entering the survey/experiment, particularly in the case of non-academic studies that are not backed by an explicit IRB approval).

Hope this helps address your concerns

Best, Dario

[1] http://meta.wikimedia.org/wiki/Research_talk:Subject_recruitment [2] http://meta.wikimedia.org/wiki/Research:Committee/Areas_of_interest/Subject_... [3] http://meta.wikimedia.org/wiki/Template:WMF-support [4] http://meta.wikimedia.org/wiki/Research_talk:Anonymity_and_conformity_over_t...

On Mar 18, 2012, at 3:40 AM, En Pine wrote:

I have concerns about this survey. I will address one set of comments to Audrey, and a second set of comments and a question to the Research Committee.

Audrey: thanks for your interest in Wikipedia. I suggest that you look at the other research that has previously discussed motivations of Wikipedia contributors and factors that can effect that motivation, such as http://meta.wikimedia.org/wiki/Wikipedia_Editors_Survey_2011 and http://meta.wikimedia.org/wiki/Research:Wikimedia_Summer_of_Research_2011/Su.... On your research Meta page, I disagree with your characterization of extant literature as “lacking,” because while it isn’t comprehensive it also shouldn’t be dismissed. Also, I am wondering why you would use a 2006 source for information about Wikipedia user contribution activity because 2006 was a long time ago in the context of Wikipedia’s lifetime. Regarding surveys of Wikipedians in general, I am skeptical about the reliability of surveys in measuring the motivations of Wikipedia contributors because so many people are not the kind of dedicated volunteer who would be likely to read Research-l or volunteer ten minutes of their time to participate in a study about their motives. Also, you will need to consider bad actors like vandals, spammers, POV pushers, and PR manipulators. Your survey might reveal interesting characteristics of certain classes of editors, but I would be very surprised if your survey results were representative of the entire population of Wikipedia editors. Another complicating factor is that motivations of any single editor can change over time. Finally, and perhaps most significantly, I have some procedural concerns. Did you discuss your survey with anyone in the Wikipedia research community before you announced it here? Your page on Meta says that you “will also request the Research Committee's support in recruiting subjects.” Your section on “Wikimedia Policies, Ethics, and Human Subjects Protection” says nothing about consultation with or approval of the Research Committee, and the most recent published minutes from the Research Committee (that I was able to find) don’t appear to show that your research was discussed by them. I think that they might have had valuable ideas that could have helped you in designing your survey and understanding the existing work on editor motivation. It is my understanding that Research Committee approval is required before soliciting Wikipedia subjects for surveys (see my question below).

RCOM members: I would appreciate an official reply to the following concerns. Is it policy that surveys which recruit participants (instead of passively examining editor contributions) must be approved by RCOM before they are sent to Wikimedia mailing lists and/or announced to the broader Wikimedia community (beyond a relatively limited scope such as a single wikiproject, such as GOCE on EN, which might give its approval to the survey only within the scope of that wikiproject)? I am under the impression from the December 12, 2011 RCOM meeting minutes that RCOM approval is required for surveys such as the one that Audrey made. My personal view is that surveyors should get RCOM’s approval before making broad public announcements which recruit research participants, because even well intended researchers can experience difficulties due to questionable assumptions built into the design a study, a limited understanding about the Wikipedia community, or a lack of knowledge about significant existing research. Also, there can be privacy and copyright concerns regarding survey data, and those reasons alone seem sufficient to require that RCOM’s approval is necessary in addition to the approval of any academic institution that is associated with a survey. Also, I am under the impression that permission from the WMF’s legal department is required, in addition to RCOM approval. In the absence of RCOM approval and WMF Legal approval, should information about such an unapproved survey be removed from Meta?

Thanks,

Pine

From: Audrey Abeyta Sent: Saturday, 17 March, 2012 13:01 To: wiki-research-l@lists.wikimedia.org Subject: [Wiki-research-l] Motivations to Contribute to Wikipedia

Hello all,

I am an undergraduate student at the University of California, Santa Barbara, conducting a senior honors thesis on users' motivations to contribute to Wikipedia. A more detailed description of the project can be read here: http://meta.wikimedia.org/wiki/Research:Motivations_to_Contribute_to_Wikiped...

My project's success is dependent on the valuable responses of Wikipedia contributors, which I am collecting through an online questionnaire. This brief questionnaire is completely anonymous and should take approximately 10 minutes to complete. If any of you are willing to complete this questionnaire, it can be accessed here: https://us1.us.qualtrics.com/SE/?SID=SV_8ixU9RkozemzC4s.

Please feel free to contact me with any questions or concerns.

Thank you in advance for your help!

Sincerely,

Audrey Abeyta

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Jodi,

you're right, we should make this documentation more visible and accessible to researchers. It's currently linked from the landing page of the Research Index as well as from the FAQ, any help to improve the documentation is very welcom. WP:BOLD ;)

Dario

On Mar 19, 2012, at 1:37 AM, Jodi Schneider wrote:

Hi Dario & all,

Thanks for that!

RCom review is still confusing to me (and I assume to many). I'm not sure how new researchers find out about the need for it (or even about the meta documentation procedures). I think more information/publicity about this could help. One possible and partial approach would be to have a regular (monthly?) summary of and reminder about RCom review sent to this email list.

-Jodi

On Mon, Mar 19, 2012 at 6:52 AM, Dario Taraborelli dtaraborelli@wikimedia.org wrote:So to briefly answer Pine's questions:

• yes, going through RCom review is the standard procedure we expect all proposals to comply with
• no, a proposal should not be removed from Meta if it hasn't been reviewed, it should only be flagged as pending review using the WMF-support template. This also means that Audrey fully complied with the expected procedure to submit a SR request.
• yes, there are privacy concerns, and this is the reason why we take the review of data collection/retention/licensing terms in the proposal very seriously. As these surveys do not fall under the WMF's privacy policy they are not reviewed by WMF Legal team unless they are considered potentially abusive. The only privacy terms that apply are those displayed on the landing page of a survey and our goal is to support best practices in setting up these terms (for example, by making sure that these terms are explicitly accepted by the participants before entering the survey/experiment, particularly in the case of non-academic studies that are not backed by an explicit IRB approval).

---

Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Dear Audrey,

On 03/19/2012 08:43 PM, emijrp wrote:

On WikiPapers we have some publications about that subject http://wikipapers.referata.com/wiki/Motivations

Regarding: "Extant literature on this topic is lacking".

As you see Emijrp has collected 10 papers categorized under motivation in his WikiPapers, and in the recently announce Wikilit group we have also many papers on motivation (mostly collected by the four other members of the Wikilit group), see e.g., this category with 30 papers:

http://wikilit.referata.com/wiki/Category:Contributor_motivation

Also the parent category:

http://wikilit.referata.com/wiki/Category:Antecedents_of_participation

52 papers are under this category. There is room for an entire review on its own. :-)

Cheers, Finn

2012/3/17 Audrey Abeyta <audrey.abeyta@gmail.com mailto:audrey.abeyta@gmail.com>

Hello all,

I am an undergraduate student at the University of California, Santa
Barbara, conducting a senior honors thesis on users' motivations to
contribute to Wikipedia. A more detailed description of the project
can be read here:
http://meta.wikimedia.org/wiki/Research:Motivations_to_Contribute_to_Wikipedia

My project's success is dependent on the valuable responses of
Wikipedia contributors, which I am collecting through an online
questionnaire. This brief questionnaire is completely anonymous and
should take approximately 10 minutes to complete. If any of you are
willing to complete this questionnaire, it can be accessed here:
https://us1.us.qualtrics.com/SE/?SID=SV_8ixU9RkozemzC4s.

Please feel free to contact me with any questions or concerns.

Thank you in advance for your help!

Sincerely,

Audrey Abeyta


_______________________________________________
Wiki-research-l mailing list
Wiki-research-l@lists.wikimedia.org
<mailto:Wiki-research-l@lists.wikimedia.org>
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l