Kerry Raymond wrote:
... we could use big data to try to pro-actively find patterns of undesirable behaviour....
I agree, including with the specific examples given, most if not all of which could be implemented within a general accuracy review system. Please consider the three-level (actually four-level, if you count the community) review system depicted at http://i.imgur.com/NhvyfXc.png
It can be gamed to secure an unfair advantage if the reviewer reputation database (which includes reviewer identity) is open, but if if the identities, reputation measurements, and algorithms are kept confidential, then it can be very robust against several kinds of bias, vandalism, incompetence, and other quality deficits. That is in part because of the same reasons which make disclosing reviewer identity less accurate.
I am trying to figure out how the reviewer reputation database can be audited without disclosing reviewer identities. It's very difficult to imagine the release any information about the reviewers and their actions, even in aggregate, without exposing information which could potentially be used to game an unfair advantage, but if the identities are double-blinded and other related information is coded, it's still possible to audit what seem be some fairly important aspects of the system's operation without any obvious vulnerabilities.
Cost is a huge part of the system's operation, and there are so many aspects of cost that need to be measured in many different ways. The system should be open to volunteers as well as paid reviewers in order to make sure it has throughput guarantees, but setting the payment schedule is difficult in an international context where the cost of living varies so widely. My initial set of paid reviewers will be around 96% in the US if I remember right.
If anyone wants to try the same thing simultaneously, please do. Eliminating duplication of effort would be a nice problem to have down the road.
Well, we could take a leaf out of the enemy's playbook and advertise on Freelancer (and similar sites) for someone to do paid reviewing of Wikipedia edits. Given the healthy trade on those sites seeking people to write Wikipedia articles (often including phrases like "and ensure it is not deleted"), it would appear to be a rich source of Wiki-savvy talent. Poacher turned gamekeeper?
And, keep any eye out for any new articles on 7 radio stations in Northern California (most of these seeking paid articles are smart enough not to mention the subject matter but this advertiser wasn't).
Kerry
-----Original Message----- From: wiki-research-l-bounces@lists.wikimedia.org [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of James Salsman Sent: Friday, 14 August 2015 1:47 PM To: wiki-research-l@lists.wikimedia.org Cc: Maribel Acosta maribelacosta21@gmail.com; Fabian Flöck f.floeck@gmail.com Subject: Re: [Wiki-research-l] "identity disclosure hurt the reliability of review systems, but not necessarily efforts provision"
Kerry Raymond wrote:
... we could use big data to try to pro-actively find patterns of undesirable behaviour....
I agree, including with the specific examples given, most if not all of which could be implemented within a general accuracy review system. Please consider the three-level (actually four-level, if you count the community) review system depicted at http://i.imgur.com/NhvyfXc.png
It can be gamed to secure an unfair advantage if the reviewer reputation database (which includes reviewer identity) is open, but if if the identities, reputation measurements, and algorithms are kept confidential, then it can be very robust against several kinds of bias, vandalism, incompetence, and other quality deficits. That is in part because of the same reasons which make disclosing reviewer identity less accurate.
I am trying to figure out how the reviewer reputation database can be audited without disclosing reviewer identities. It's very difficult to imagine the release any information about the reviewers and their actions, even in aggregate, without exposing information which could potentially be used to game an unfair advantage, but if the identities are double-blinded and other related information is coded, it's still possible to audit what seem be some fairly important aspects of the system's operation without any obvious vulnerabilities.
Cost is a huge part of the system's operation, and there are so many aspects of cost that need to be measured in many different ways. The system should be open to volunteers as well as paid reviewers in order to make sure it has throughput guarantees, but setting the payment schedule is difficult in an international context where the cost of living varies so widely. My initial set of paid reviewers will be around 96% in the US if I remember right.
If anyone wants to try the same thing simultaneously, please do. Eliminating duplication of effort would be a nice problem to have down the road.
_______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
wiki-research-l@lists.wikimedia.org