Kerry Raymond wrote:
... we could use big data to try to pro-actively find patterns of undesirable behaviour....
I agree, including with the specific examples given, most if not all of which could be implemented within a general accuracy review system. Please consider the three-level (actually four-level, if you count the community) review system depicted at http://i.imgur.com/NhvyfXc.png
It can be gamed to secure an unfair advantage if the reviewer reputation database (which includes reviewer identity) is open, but if if the identities, reputation measurements, and algorithms are kept confidential, then it can be very robust against several kinds of bias, vandalism, incompetence, and other quality deficits. That is in part because of the same reasons which make disclosing reviewer identity less accurate.
I am trying to figure out how the reviewer reputation database can be audited without disclosing reviewer identities. It's very difficult to imagine the release any information about the reviewers and their actions, even in aggregate, without exposing information which could potentially be used to game an unfair advantage, but if the identities are double-blinded and other related information is coded, it's still possible to audit what seem be some fairly important aspects of the system's operation without any obvious vulnerabilities.
Cost is a huge part of the system's operation, and there are so many aspects of cost that need to be measured in many different ways. The system should be open to volunteers as well as paid reviewers in order to make sure it has throughput guarantees, but setting the payment schedule is difficult in an international context where the cost of living varies so widely. My initial set of paid reviewers will be around 96% in the US if I remember right.
If anyone wants to try the same thing simultaneously, please do. Eliminating duplication of effort would be a nice problem to have down the road.