This kind of problem will arise from all systems where an asymmetry is
introduced. One part can then fight the other part and win out, due tor
the ordering of the fight. The problem is rather difficult to solve as
the system must be symmetrical not only between two consecutive edits
but edits split out over several postings, merged, intermixed with other
edits etc.
John E
Luca de Alfaro skrev:
Daniel is making some very good points.
Our current algorithm is vulnerable to two kinds of attacks:
* Sock puppets
* People who split an edit into many smaller ones, done with sock
puppets or not, in order to raise the trust of text.
We think we know how to fix or at least mitigate both problems. This
is why I say that a "real-time" system that colors revisions as they
are made is a couple of months (I hope) away. The challenge is not so
much to reorganize the code to work from wikipedia dumps to real-time
edits. The challenge for us is to analyze, implement, and quantify
the performance of versions of the algorithms that are resistant to
attack. For those of you who have checked our papers, you would have
seen that not only we propose algorithms, but we do extensive
performance studies on how good the algorithms are. We will want to
do the same for the algorithms for fighting sock puppets.
About the proposal by Daniel: time alone does not cover our full set
of concerns.
I can every day use identity A to erase some good text, and identity B
to put it back in. Then, the reputation of B would grow a bit every
day, even though B did not do much effort.
We are thinking of some other solutions... but please forgive us for
keeping this to ourselves a little bit longer... we would like to have
a chance to do a full study before shooting our mouths off...
Luca
On Dec 19, 2007 6:05 PM, Daniel Arnold <arnomane(a)gmx.de
<mailto:arnomane@gmx.de>> wrote:
Hello Luca,
1. The trust coloring rightly colored orange
(low-trust) some
unreliable content,
Yes I was lost in translation. ;-)
2. and the Wikipedia people were quick in
reverting it.
Yes.
Note that we also highlight as low trust text
that is by anonymous
contributors. The text will then gain trust as it is revised.
One possible weakness came into my mind after I also read your
paper. Your
algorithm is perhapes a bit vulnerable to "sock puppets". Imagine
person A
with one account and person B with two accounts. Both have a medium
reputation value for their accounts. User A edits an article with
his account
4 times. All 4 subsequent edits are taken together and the article
has a
maximum trust value according to the user's reputation. User B
makes as well
4 edits to an article but switches between his accounts and thus
"reviews"
his own edits. If I understand your algorithm correctly the sock
puppeted
article is trusted more than the other one.
Quite some time ago I reflected how to avoid incentives for sock
puppets in
karma systems without even knowing which accounts are sock puppets:
http://meta.wikimedia.org/wiki/Meritokratischer_Review (sadly in
German ;-).
The system described there differs from your approach but the idea
on how to
avoid incentives for sock puppets without even knowing who a sock
puppet is
could perhapes adapted to your system.
The basic idea for a sock puppet proof metric is is that a person
has only a
limited amount of time for editing (I don't consider bots cause
they are
easily detectable by humans). A single person needs the same time
for e.g. 4
edits (in the following I assume each edit has the same length in
bytes)
regardless how much accounts are used but two different people
with each 2
edits only need half of the (imaginary) time (you don't need to
measure any
time untits at all).
So the maximum possible reliability person B can apply to the
article with its
two accounts (let us say each acount has 2 edits = 4 total edits)
has to be
the same as the one which is possible with person A's single
account (4
edits). So in general two accounts with each X edits should never
be able to
add more trust to an article than one person with 2*X edits (note:
edit count
number is only for illustration, you can take another appropriate
contribution unit).
About 2, I am very glad that bad edits are
quickly reverted;
this is the
whole reason Wikipedia has worked up to now.
Still, it might be easier for editors to find content to check
via the
coloring, rather than by staring at diffs.
That's certainly true for articles not on your watchlist (or bad
edits that
were forgotten and are still the latest version).
- Finding when flagged revisions are out of
date (there may be
a new
high-trust version later)
Well as I said I'd love to see flagged revisions and your system
combined (in
a way described by my previous mail). An automated system probably
always has
some weaknesses some clever people can abuse but it is very fast,
while a
hand crafted system depends on the speed of individual persons but
is much
harder to fool.
BTW, as the method is language-independent, we
look forward to
doing the
same for wikipedias in other languages.
Good to know. :-)
Arnomane
_______________________________________________
Wikiquality-l mailing list
Wikiquality-l(a)lists.wikimedia.org
<mailto:Wikiquality-l@lists.wikimedia.org>
http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
------------------------------------------------------------------------
_______________________________________________
Wikiquality-l mailing list
Wikiquality-l(a)lists.wikimedia.org
http://lists.wikimedia.org/mailman/listinfo/wikiquality-l