Hello Luca,
1. The trust coloring rightly colored orange
(low-trust) some
unreliable content,
Yes I was lost in translation. ;-)
2. and the Wikipedia people were quick in reverting
it.
Yes.
Note that we also highlight as low trust text that is
by anonymous
contributors. The text will then gain trust as it is revised.
One possible weakness came into my mind after I also read your paper. Your
algorithm is perhapes a bit vulnerable to "sock puppets". Imagine person A
with one account and person B with two accounts. Both have a medium
reputation value for their accounts. User A edits an article with his account
4 times. All 4 subsequent edits are taken together and the article has a
maximum trust value according to the user's reputation. User B makes as well
4 edits to an article but switches between his accounts and thus "reviews"
his own edits. If I understand your algorithm correctly the sock puppeted
article is trusted more than the other one.
Quite some time ago I reflected how to avoid incentives for sock puppets in
karma systems without even knowing which accounts are sock puppets:
http://meta.wikimedia.org/wiki/Meritokratischer_Review (sadly in German ;-).
The system described there differs from your approach but the idea on how to
avoid incentives for sock puppets without even knowing who a sock puppet is
could perhapes adapted to your system.
The basic idea for a sock puppet proof metric is is that a person has only a
limited amount of time for editing (I don't consider bots cause they are
easily detectable by humans). A single person needs the same time for e.g. 4
edits (in the following I assume each edit has the same length in bytes)
regardless how much accounts are used but two different people with each 2
edits only need half of the (imaginary) time (you don't need to measure any
time untits at all).
So the maximum possible reliability person B can apply to the article with its
two accounts (let us say each acount has 2 edits = 4 total edits) has to be
the same as the one which is possible with person A's single account (4
edits). So in general two accounts with each X edits should never be able to
add more trust to an article than one person with 2*X edits (note: edit count
number is only for illustration, you can take another appropriate
contribution unit).
About 2, I am very glad that bad edits are quickly
reverted; this is the
whole reason Wikipedia has worked up to now.
Still, it might be easier for editors to find content to check via the
coloring, rather than by staring at diffs.
That's certainly true for articles not on your watchlist (or bad edits that
were forgotten and are still the latest version).
- Finding when flagged revisions are out of date
(there may be a new
high-trust version later)
Well as I said I'd love to see flagged revisions and your system combined (in
a way described by my previous mail). An automated system probably always has
some weaknesses some clever people can abuse but it is very fast, while a
hand crafted system depends on the speed of individual persons but is much
harder to fool.
BTW, as the method is language-independent, we look
forward to doing the
same for wikipedias in other languages.
Good to know. :-)
Arnomane