This comment is very interesting, and it points out that we most likely are using too dark shades of orange (this is customizable in php btw). We have 10 equally-spaced shades of orange, from the darkest (trust 0) to pure white (trust 9). According to the current coloring scheme, even text with trust 7, which has been revised etc, gets some visible (two steps down from pure white) shade of orange. It might be visually better to have 8,9 both as pure white, and lighten the shade of levels 6,7.
In the new codebase, we have increased the speed at which text gains trust (we noticed also it was a bit too orange in that old demo). The vote button also helps text gain trust much more quickly, as people can just click there to validate the text, rather than having to do an edit. People can only raise the trust of text up to their own reputation level (which also goes from 0 to 9), so that spammers cannot enter an edit, then use sock-puppets to make the orange coloring disappear.
Yes, we did compare the results with a trust system based purely on text age; see Figure 7 of http://www.soe.ucsc.edu/~luca/papers/08/wikisym08-trust.html On the Wikipedia, people are so dedicated that most pages are visited regularly, so age of text is a good indicator of text quality. Using a reputation system as we do enables us to assign medium trust (trust level 5) to the brand-new contributions by high-reputation authors (which means, in practice, anyone with a moderate history of good edits). If you go simply by text age, then you would end up assigning low trust to these contributions when they are brand new. Thus, the use of a reputation system enables us to assign _more_ trust to text. This is is why Figure 7 shows that the trust based on a reputation system is more precise: low trust is assigned more sparingly, and it is a more precise predictor of future deletions.
In summary, on the Wikipedia the use of a reputation system enables us to:
- Assign more trust to new text by good reputation authors - Make it hard for spammers to cause their contributions to become fully trusted.
As Gregory points out, though, at least before the introduction of the vote button, our coloring had too much text in the lighter shades of orange. Whether this would remain a problem even after people can vote for the correctness of text, I don't know. We believe the vote button is very useful in enabling good text gain trust quickly.
Also note that in many less-followed wikis than the Wikipedia, many pages remain unchecked for relatively long periods, so using text age there would not necessarily work well -- but I don't have data on hand to back this claim.
Luca
These performance metrics are better than I would have guessed from browsing through the output. How does the color mapping reflect the trust values? Basically when I use it I see a *lot* of colored things which are perfectly fine. At least for me, the difference between shades is far less cognitively significant than colored vs non-colored, so that may be the source of my confusion.
Have you compared your system to a simple toy trust metric? I'd propose "revisions by users in their first week and before their first 7 (?) edits are untrusted". This reflects the existing automatic trust system on the site (auto-confirmation), and also reflects the a type of trust checking applied manually by editors. I think thats the bar any more sophisticated trust metric needs to outperform.
Thank you so much for your response! _______________________________________________ Wikipedia-l mailing list Wikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikipedia-l