Hi Linxuan,
Thank you for your question:
... What does the "reputation score" in the description refer to?
I've asked Priyanka to reply with her current design, but here is some of the advice I gave her:
"Each reviewer needs, at a minimum, data indicating the number and proportion of reviewers who have agreed with them. However, the third level of tie-breaking review introduces an extra bit for each disagreement which determines whether agreement or disagreement should be counted in their favor. So, even if a given reviewer only agrees with 50% of the other reviewers, the determination of the tie breaker in each case of disagreement controls whether their reputation score ranges from 0% to 100%. (As too does the agreement proportion, which is unlikely to be exactly 50%.)
"Do you want the reviewers to know their agreement ratios and reputation scores? How might their behavior change if they are and aren't told those? Could there ever be a case when you might want to withhold them? Would there ever be a benefit from distorting them? How about displaying them as a range instead of distorting or withholding them? That last possibility seems superior to me. You might want to do that when you are unsure that the precision of the mathematical values is near the accuracy of the knowledge they represent. Do you want to be able to tell each reviewer the responses which have contributed to defects in their reputation scores, i.e., do you want them to know which disagreements were tie-broken against their favor?"
Her reply at the time was:
"In case of two reviewers agreeing, we add a +1 to the reputation. In case of disagreement, we seek the opinion of a 3rd reviewer. If A says Yes, B says No and C says Yes to an edit, A and C will have an agreement ratio of 50% and reputation of 100%, whereas B will have an agreement ratio of 0 and reputation 0%? This would of course change as more edits are reviewed by them."
I believe that is still an accurate description of the current design.
Finally, I regret that the GSoC program doesn't allow more than one student per
Best regards, Jim Salsman