Well, we could take a leaf out of the enemy's playbook and advertise on Freelancer
(and similar sites) for someone to do paid reviewing of Wikipedia edits. Given the healthy
trade on those sites seeking people to write Wikipedia articles (often including phrases
like "and ensure it is not deleted"), it would appear to be a rich source of
Wiki-savvy talent. Poacher turned gamekeeper?
And, keep any eye out for any new articles on 7 radio stations in Northern California
(most of these seeking paid articles are smart enough not to mention the subject matter
but this advertiser wasn't).
Kerry
-----Original Message-----
From: wiki-research-l-bounces(a)lists.wikimedia.org
[mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of James Salsman
Sent: Friday, 14 August 2015 1:47 PM
To: wiki-research-l(a)lists.wikimedia.org
Cc: Maribel Acosta <maribelacosta21(a)gmail.com>om>; Fabian Flöck
<f.floeck(a)gmail.com>
Subject: Re: [Wiki-research-l] "identity disclosure hurt the reliability of review
systems, but not necessarily efforts provision"
Kerry Raymond wrote:
...
we could use big data to try to pro-actively find patterns of undesirable behaviour....
I agree, including with the specific examples given, most if not all of which could be
implemented within a general accuracy review system.
Please consider the three-level (actually four-level, if you count the
community) review system depicted at
http://i.imgur.com/NhvyfXc.png
It can be gamed to secure an unfair advantage if the reviewer reputation database (which
includes reviewer identity) is open, but if if the identities, reputation measurements,
and algorithms are kept confidential, then it can be very robust against several kinds of
bias, vandalism, incompetence, and other quality deficits. That is in part because of the
same reasons which make disclosing reviewer identity less accurate.
I am trying to figure out how the reviewer reputation database can be audited without
disclosing reviewer identities. It's very difficult to imagine the release any
information about the reviewers and their actions, even in aggregate, without exposing
information which could potentially be used to game an unfair advantage, but if the
identities are double-blinded and other related information is coded, it's still
possible to audit what seem be some fairly important aspects of the system's operation
without any obvious vulnerabilities.
Cost is a huge part of the system's operation, and there are so many aspects of cost
that need to be measured in many different ways. The system should be open to volunteers
as well as paid reviewers in order to make sure it has throughput guarantees, but setting
the payment schedule is difficult in an international context where the cost of living
varies so widely. My initial set of paid reviewers will be around 96% in the US if I
remember right.
If anyone wants to try the same thing simultaneously, please do.
Eliminating duplication of effort would be a nice problem to have down the road.
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l