Neil Harris wrote:
You should be able to do the whole thing in one go.
It's a bit like clocks; accurate clocks are defined to be those which tend to give similar times to other accurate clocks. Inaccurate clocks do not have this property.
With standards now depending on atomic clocks we now need to to add the occasional leap second to ensure that the earth behaves as it should in its journey around the sun. How would we clean the raters' clock?
In this case, good raters are defined to be those who give ratings which tend to correlate well with true ratings, which are in turn extracted by ratings given by other good raters. Even though this is necessarily a recursive definition, it can still be used to generate a tractable set of simultaneous equations.
"True" ratings being a matter of subjectivity, and people being people, there may also be more than one mutually-coherent cluster of raters. If you're worried about active coordinated attacks by ratings spammers, or want to try to average across political viewpoints, you can seed things with a core of users known to be likely to be both good and impartial raters.
Proceeding in this vain [sic!], we are ultimately led to the great Rooto-Rater, and that problem is theological. :-)
Ec