wjhonson@aol.com wrote:
With the right hand, you rate the raters. So each of us gets a clue stick and goes around whacking good editors "good rater" rate up a notch by voting for them as raters.
With the left hand, you rate the articles, and when other editors agree with you, they whack you and your "good rater" score goes up.
Now with the giant nose of Zenobia, you multiply the article rating by the raters rating, and average.
Thusly and so, articles get a good rating based on the best raters rating them good, and nasty bad evil raters, ratings fall into the first circle (i.e. they are weighted as nothing).
Will Johnson
You should be able to do the whole thing in one go.
It's a bit like clocks; accurate clocks are defined to be those which tend to give similar times to other accurate clocks. Inaccurate clocks do not have this property.
In this case, good raters are defined to be those who give ratings which tend to correlate well with true ratings, which are in turn extracted by ratings given by other good raters. Even though this is necessarily a recursive definition, it can still be used to generate a tractable set of simultaneous equations.
"True" ratings being a matter of subjectivity, and people being people, there may also be more than one mutually-coherent cluster of raters. If you're worried about active coordinated attacks by ratings spammers, or want to try to average across political viewpoints, you can seed things with a core of users known to be likely to be both good and impartial raters.
-- Neil