-------- Message original --------
Sujet: Re: [Wiki-research-l] Experimental study of informal rewards in peer production
Date : Thu, 26 Apr 2012 15:50:44 -0400
De : Michael Restivo <mike.restivo(a)gmail.com>
Pour : Chitu Okoli <Chitu.Okoli(a)concordia.ca>ca>, Research into Wikimedia content and
communities <wiki-research-l(a)lists.wikimedia.org>
Hi Chitu,
Yes, your conjecture is spot-on. Here is a more detailed response that I sent to Joseph. I
tried sending this to the wiki-research-l but the email keeps bouncing back to me. If
you're interested and willing to share it with the list, that would be acceptable to
me.
We thought about this question quite extensively and there are a few reasons why we
sampled the top 1% (which we didn't get around to discussing in this brief paper).
First, because of the high degree of contribution inequality in Wikipedia's editing
community, we were primarily interested in how status rewards affect the all-important
core of highly-active editors. There is also a lot of turn-over in the long tail of the
distribution, and even among the most active editors, there is considerable heterogeneity.
Focusing on the most active users ensured us sufficient statistical power. (Post-hoc power
analysis suggests that our sample size would need to be several thousand users in the
80-90th percentiles, and several hundred in the 90-99th percentiles, to discern an effect
of the same strength.) Also, we considered the question of construct validity: which
users are deserving (so to speak) of receiving an editing award or social recognition of
their work?
You are right that it should be fairly easy to extend this analysis beyond just the
top 1%, but just how wide a net to cast remains a question. The issue of power calculation
and sample size becomes increasingly difficult to manage for lower deciles because of the
power-law distribution. And I don't think it would be very meaningful to assess the
effect of barnstars on the bottom half of the distribution, for example, for the
substantive reasons I mentioned above. Still, I'd be curious to hear what you think,
and whether there might be some variations on this experiment that could overcome these
limitations.
In terms of data dredging, that is always a concern and I completely understand where
you are coming from. In fact, as both and author and consumer of scientific knowledge,
I'm rarely ever completely satisfied. For example, a related concern that I have is
the filing cabinet effect - when research produces null (or opposite) results and hence
the authors decide to not attempt to have it published.
In this case, I actually started this project with the hunch that barnstars would lead
to a slight decline in editing behavior; my rationale was that rewards would act as social
markers that editors' past work was sufficient to earn social recognition and hence
receiving such a reward would signal that the editor had "done enough" for the
time being. In addition to there being substantial support for this idea in the economics
literature, this intuition stemmed from hearing about an (unpublished) observational study
of barnstars by Gueorgi Kossinets (formerly at Cornell, now at Google) that suggested
editors receive barnstars at the peak of their editing activity. Of course, we chose an
experimental design precisely to help us to tease out the causal direction as well as what
effect barnstars have for recipients relative to their unrewarded counterparts. We felt
like no matter what we found - either a positive, negative, or even no effect - it would
have been interesting
enough to publish, so hopefully that alleviates some of your concerns.
Please let me know if you have any other questions, and I'd love to hear your
thoughts about potential follow-ups to this research.
Regards,
Michael
On Thu, Apr 26, 2012 at 3:30 PM, Chitu Okoli <Chitu.Okoli(a)concordia.ca
<mailto:Chitu.Okoli@concordia.ca>> wrote:
One obvious issue is that it would be unethical to award barnstars to contributors who
did not deserve them. However, the 1% most productive contributors, by definition,
deserved the barnstars that the experimenter awarded them. Awarding barnstars to
undeserving contributors for experimental purposes probably would not have flown so easily
by the ethical review board. As the article notes:
----------
This study's research protocol was approved by the Committees on Research
Involving Human Subjects (IRB) at the State University of New York at Stony Brook (CORIHS
#2011-1394). Because the experiment presented only minimal risks to subjects, the IRB
committee determined that obtaining prior informed consent from participants was not
required.
----------
This is my conjecture; I'd like to hear the author's comments.
~ Chitu
-------- Message original --------
Sujet: [Wiki-research-l] Experimental study of informal rewards in peer production
De : Joseph Reagle <joseph.2011(a)reagle.org
<mailto:joseph.2011@reagle.org>>
Pour : michael.restivo(a)stonybrook.edu <mailto:michael.restivo@stonybrook.edu>
Copie à : Research into Wikimedia content and communities
<wiki-research-l(a)lists.wikimedia.org
<mailto:wiki-research-l@lists.wikimedia.org>>
Date : 26 Avril 2012 11:42:01
In this
[
study](http://www.plosone.org/article/info:doi%2F10.1371%2Fjournal.pone.003…:
Abstract: We test the effects of informal rewards in
online peer production. Using a randomized, experimental design, we assigned editing
awards or “barnstars” to a subset of the 1% most productive Wikipedia contributors.
Comparison with the control group shows that receiving a barnstar increases productivity
by 60% and makes contributors six times more likely to receive additional barnstars from
other community members, revealing that informal rewards significantly impact individual
effort.
I wonder why it is limited to the top 1%? I'd love to see the analysis repeated
(should be trivial) on each decile. Besides satisfying my curiosity, some rationale and/or
discussion of other deciles would also address any methodological concern about data
dredging.
--
Michael Restivo
Department of Sociology
Social and Behavioral Sciences S-433
Stony Brook University
Stony Brook, NY 11794
mike.restivo(a)gmail.com <mailto:mike.restivo@gmail.com>