Did the paper look at the other side of the coin where an anonymous reviewer uses their
dislike of someone to give them a poorer score than they deserved? From the description of
the experiment, I assume none of the participants knew one another before the experiment
and so could not have brought a dislike into the room so that issue was presumably not
tested.
Generally in reviewing there are 3 roles involved:
* The “owner” of the thing being reviewed
* The “reviewer”
* The “coordinator” of the reviewing
In blind-to-owner reviewing, the coordinator (and sometimes other reviewers) know the
identity of the reviewer even if the owner does not. To some extent, this serves to keep
the reviewers honest in relation to any conflict of interest with the owner by either
reviewing scrupulously fairly or by refusing to review. Obviously in smaller communities
(e.g. a niche academic topic), where real names are used and there is a strong chance of
the coordinator (or other reviewers) knowing about your likes and dislikes, there is a
stronger likelihood of the reviews being honest or being refused. This means a small
number of reviews can suffice as a reasonable opinion as to the merits of the paper. In a
large community, like TripAdvisor which uses usernames and which is not a blind review (in
the sense that the user names are disclosed but not real names), it is far less likely
that conflicts of interest will be detected so a “reasonable opinion” requires a lot more
reviews to hope to dominate any bias introduced by friends/foes acting as reviewers.
With respect to Wikipedia, where everyone is a potential reviewer (and executioner) of
every edit and we have watchlists which allow individuals to choose to be reviewers of
particular articles (making them the most likely reviewers of edits of that article). As a
result, we often have few “reviews” and an “enemy” can easily revert an edit. An enemy can
use a user’s contributions to easily find other edit’s and revert them too (if some flimsy
pretext exists), allowing harassment (a pattern of behaviour that may be very visible to
the victim but not to the larger community). We really only have a “negative review”
system in Wikipedia. If a reviewer feels positive about an edit, then they usually leave
it alone. They can leave a thanks or Wikilove or a positive Talk message (but the stats
would suggest this is rarely done), but that any of those positive actions does not
prevent a revert. Oh, and we do it all behind anonymity and pseudonyms with a policy
against outing ensuring there is little real world accountability for people’s actions.
Is it any wonder we have a toxic culture on Wikipedia? Build a housing estate with dark
alleyways with no visibility to passersby and you create a mugger’s paradise and a no-go
zone for everyone else. Build a collaborative environment online, create lots of dark
corners, remove real world accountability, have a relatively ineffective solutions to
sockpuppeting and we get the ultimate playground for the cyber bully. Oh, and we’ve just
designed a Visual Editor to funnel more newcomers into the dark corners.
I think we need to do a lot of work on Wikipedia’s design for a good culture. But in
meantime (knowing that hell freezes over faster than changes on Wikipedia), we could use
big data to try to pro-actively find patterns of undesirable behaviour. If one account
experiences a high level of reverts from another account, maybe someone else should be
reviewing those edits to see if it’s bullying or vandalism. If one account appears to be
doing a lot of reverting, maybe someone else should be reviewing to see if there’s some
kind of undesirable pattern to it. At the moment, we do not appear to be proactive. We
rely on victims knowing how to complain and how to provide a lot of documentation to
support their case. This is something newcomers are unlikely to know how to do; indeed,
even more experienced editors have been known to just walk away than put up with
aggressive behaviour.
Kerry
From: wiki-research-l-bounces(a)lists.wikimedia.org
[mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Kevin G Crowston
Sent: Friday, 14 August 2015 6:55 AM
To: wiki-research-l(a)lists.wikimedia.org
Subject: Re: [Wiki-research-l] "identity disclosure hurt the reliability of review
systems, but not necessarily efforts provision"
Subject: Wiki-research-l Digest, Vol 120, Issue 16
Can someone who has access to that paper please share the method and results as fair use?
Here are a few key paragraphs from the methods and results section.
Method
"We test two “social conditions” Anonymous vs. Identity disclosure ( anonymous :
participants do not know each other; identity : participants were first asked to introduce
themselves before experiments and, during the experiments, participants were shown name of
their reviewer on the computer screen and vice versa) . In addition to social condition,
we introduce “repeated matching” ( 1 round : matching of counterpart and reviewer changes
every round; 3 round : each participant will be matched with the same counterpart and the
same reviewer for three consecutive rounds) as another dimension to compare the effect of
short-term and long-term social pressure. Employing 2 × 2 design, our experiments consist
of four treatments, varying in two dimensions.
Parameters used are P =120, k =0.15. NE prediction is e* =10 under correct-reporting
(Proposition 1) and e* =1 under over-reporting (Proposition 2). Expected p rofit is
π *=45 and π *=59.85, respectively. Experiments were implemented using z-Tree software
(Fischbacher 2007). Participants of experiments consist of 108 undergraduate students
at a large pu blic research university in the United States. For Anonymous 1
round treatment, we conducted three experimental sessions. Each of the rest
three treatments consisted of two sessions. Each session had 12 participants with 15
decision rounds. Subjects received a course credit and cash payment based on their results
of the game."
Findings
"In Anonymous , correct-reporting is most frequently observed at 42% (1-round) and
59% (3-round). In Identity , over-reporting accounts for more than 64% and 72% ,
respectively. The findings in reporting behaviors are in line with our expectation as
shown on Propositions 1 and 2. That is, when identity is revealed, reviewers feel
social pressure to be nice toward players. Therefore, evaluation systems are
considered less reliable because it fails to reflect the true efforts.
Another finding is that reporting behavior seems st rengthened for a longer period of
matching. Repeate d matching seems to make reviewers form a stronger so cial pressure. For
example, higher proportion of over-reporting in Identity 3 round suggests an increased
social pressure due to expec tation of a long-term relationship between reviewers and
players. “
"Median efforts from four treatments are all lower than that of standard economics
prediction (assuming objective-r eporting, e* =10). This suggests that players suspect
reliability of evaluation systems. For Anonymous 1 round and 3 round where
correct-reporting is observed at a high rate, the median efforts are both 8. This high
median efforts imply that players in anonymous settings generally hold a stronger
belief that evaluation systems are still reliable at some extent.
Surprisingly, for Identity 1 round where scores are mostly inflated, the median
effort is still 8. The average effort is at 8.2, even higher than both Anonymous
treatments (7.3 and 7.6). This finding suggests a gap between players’ belief and
reviewers’ behavior. While reviewers feel strong social pressure by revealing
identity, players doubt that social pressure of reviewers will be enough to be nice.
However , such doubt disappears when players recognize a long-term relationship. For
Identity 3 round , the median and average efforts decrease into 5 and 5.6, respectively,
the lowest level among all treatments."
Kevin Crowston | Distinguished Professor of Information Science | School of Information
Studies
Syracuse University
348 Hinds Hall
Syracuse, New York 13244
t (315) 443.1676 f 315.443.5806 e <mailto:crowston@syr.edu>
mailto:crowston@syr.edu
<http://crowston.syr.edu/>
crowston.syr.edu