This comment is very interesting, and it points out that we most likely are
using too dark shades of orange (this is customizable in php btw).
We have 10 equally-spaced shades of orange, from the darkest (trust 0) to
pure white (trust 9).
According to the current coloring scheme, even text with trust 7, which has
been revised etc, gets some visible (two steps down from pure white) shade
of orange.
It might be visually better to have 8,9 both as pure white, and lighten the
shade of levels 6,7.
In the new codebase, we have increased the speed at which text gains trust
(we noticed also it was a bit too orange in that old demo). The vote button
also helps text gain trust much more quickly, as people can just click there
to validate the text, rather than having to do an edit. People can only
raise the trust of text up to their own reputation level (which also goes
from 0 to 9), so that spammers cannot enter an edit, then use sock-puppets
to make the orange coloring disappear.
Yes, we did compare the results with a trust system based purely on text
age; see Figure 7 of
http://www.soe.ucsc.edu/~luca/papers/08/wikisym08-trust.html
On the Wikipedia, people are so dedicated that most pages are visited
regularly, so age of text is a good indicator of text quality. Using a
reputation system as we do enables us to assign medium trust (trust level 5)
to the brand-new contributions by high-reputation authors (which means, in
practice, anyone with a moderate history of good edits). If you go simply
by text age, then you would end up assigning low trust to these
contributions when they are brand new. Thus, the use of a reputation system
enables us to assign _more_ trust to text. This is is why Figure 7 shows
that the trust based on a reputation system is more precise: low trust is
assigned more sparingly, and it is a more precise predictor of future
deletions.
In summary, on the Wikipedia the use of a reputation system enables us to:
- Assign more trust to new text by good reputation authors
- Make it hard for spammers to cause their contributions to become fully
trusted.
As Gregory points out, though, at least before the introduction of the vote
button, our coloring had too much text in the lighter shades of orange.
Whether this would remain a problem even after people can vote for the
correctness of text, I don't know. We believe the vote button is very
useful in enabling good text gain trust quickly.
Also note that in many less-followed wikis than the Wikipedia, many pages
remain unchecked for relatively long periods, so using text age there would
not necessarily work well -- but I don't have data on hand to back this
claim.
Luca
These performance metrics are better than I would have
guessed from
browsing through the output. How does the color mapping reflect the
trust values? Basically when I use it I see a *lot* of colored things
which are perfectly fine. At least for me, the difference between
shades is far less cognitively significant than colored vs
non-colored, so that may be the source of my confusion.
Have you compared your system to a simple toy trust metric? I'd
propose "revisions by users in their first week and before their first
7 (?) edits are untrusted". This reflects the existing automatic
trust system on the site (auto-confirmation), and also reflects the a
type of trust checking applied manually by editors. I think thats
the bar any more sophisticated trust metric needs to outperform.
Thank you so much for your response!
_______________________________________________
Wikipedia-l mailing list
Wikipedia-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikipedia-l