>
>
>
Hi Linxuan,
Thank you for your question:
>... What does the "reputation score" in the description refer to?
I've asked Priyanka to reply with her current design, but here is some
of the advice I gave her:
"Each reviewer needs, at a minimum, data indicating the number and
proportion of reviewers who have agreed with them. However, the third level
of tie-breaking review introduces an extra bit for each disagreement which
determines whether agreement or disagreement should be counted in their
favor. So, even if a given reviewer only agrees with 50% of the other
reviewers, the determination of the tie breaker in each case of
disagreement controls whether their reputation score ranges from 0% to
100%. (As too does the agreement proportion, which is unlikely to be
exactly 50%.)
"Do you want the reviewers to know their agreement ratios and reputation
scores? How might their behavior change if they are and aren't told those?
Could there ever be a case when you might want to withhold them? Would
there ever be a benefit from distorting them? How about displaying them as
a range instead of distorting or withholding them? That last possibility
seems superior to me. You might want to do that when you are unsure that
the precision of the mathematical values is near the accuracy of the
knowledge they represent. Do you want to be able to tell each reviewer the
responses which have contributed to defects in their reputation scores,
i.e., do you want them to know which disagreements were tie-broken against
their favor?"
Her reply at the time was:
"In case of two reviewers agreeing, we add a +1 to the reputation. In case
of disagreement, we seek the opinion of a 3rd reviewer. If A says Yes, B
says No and C says Yes to an edit, A and C will have an agreement ratio of
50% and reputation of 100%, whereas B will have an agreement ratio of 0
and reputation 0%? This would of course change as more edits are reviewed
by them."
I believe that is still an accurate description of the current design.
Finally, I regret that the GSoC program doesn't allow more than one student
per
Best regards,
Jim Salsman