New subject: Wikipedia colored according to trust

19 Dec 2007

------------------------------------------------------------------------
Date: Wed, 19 Dec 2007 19:44:14 -0800
From: luca@dealfaro.org
To: wikiquality-l@lists.wikimedia.org
Subject: Re: [Wikiquality-l] Wikipedia colored according to trust

Daniel is making some very good points.

Our current algorithm is vulnerable to two kinds of attacks:

    * Sock puppets
    * People who split an edit into many smaller ones, done with
      sock puppets or not, in order to raise the trust of text.

We think we know how to fix or at least mitigate both problems. 
This is why I say that a "real-time" system that colors revisions
as they are made is a couple of months (I hope) away.  The
challenge is not so much to reorganize the code to work from
wikipedia dumps to real-time edits.  The challenge for us is to
analyze, implement, and quantify the performance of versions of
the algorithms that are resistant to attack.  For those of you who
have checked our papers, you would have seen that not only we
propose algorithms, but we do extensive performance studies on how
good the algorithms are.  We will want to do the same for the
algorithms  for fighting sock puppets.

About the proposal by Daniel: time alone does not cover our full
set of concerns.
I can every day use identity A to erase some good text, and
identity B to put it back in. Then, the reputation of B would grow
a bit every day, even though B did not do much effort.
We are thinking of some other solutions... but please forgive us
for keeping this to ourselves a little bit longer... we would like
to have a chance to do a full study before shooting our mouths off...

Luca

On Dec 19, 2007 6:05 PM, Daniel Arnold <arnomane@gmx.de
<mailto:arnomane@gmx.de>> wrote:

    Hello Luca,

    >    1. The trust coloring rightly colored orange (low-trust) some
    >    unreliable content,

    Yes I was lost in translation. ;-)

    >    2. and the Wikipedia people were quick in reverting it.

    Yes.

    > Note that we also highlight as low trust text that is by
    anonymous
    > contributors. The text will then gain trust as it is revised.

    One possible weakness came into my mind after I also read your
    paper. Your
    algorithm is perhapes a bit vulnerable to "sock puppets".
    Imagine person A
    with one account and person B with two accounts. Both have a
    medium
    reputation value for their accounts. User A edits an article
    with his account
    4 times. All 4 subsequent edits are taken together and the
    article has a
    maximum trust value according to the user's reputation. User B
    makes as well
    4 edits to an article but switches between his accounts and
    thus "reviews"
    his own edits. If I understand your algorithm correctly the
    sock puppeted
    article is trusted more than the other one.

    Quite some time ago I reflected how to avoid incentives for
    sock puppets in
    karma systems without even knowing which accounts are sock
    puppets:
    http://meta.wikimedia.org/wiki/Meritokratischer_Review (sadly
    in German ;-).
    The system described there differs from your approach but the
    idea on how to
    avoid incentives for sock puppets without even knowing who a
    sock puppet is
    could perhapes adapted to your system.

    The basic idea for a sock puppet proof metric is is that a
    person has only a
    limited amount of time for editing (I don't consider bots
    cause they are
    easily detectable by humans). A single person needs the same
    time for e.g. 4
    edits (in the following I assume each edit has the same length
    in bytes)
    regardless how much accounts are used but two different people
    with each 2
    edits only need half of the (imaginary) time (you don't need
    to measure any
    time untits at all).

    So the maximum possible reliability person B can apply to the
    article with its
    two accounts (let us say each acount has 2 edits = 4 total
    edits) has to be
    the same as the one which is possible with person A's single
    account (4
    edits). So in general two accounts with each X edits should
    never be able to
    add more trust to an article than one person with 2*X edits
    (note: edit count
    number is only for illustration, you can take another appropriate
    contribution unit).

    > About 2, I am very glad that bad edits are quickly reverted;
    this is the
    > whole reason Wikipedia has worked up to now.
    > Still, it might be easier for editors to find content to
    check via the
    > coloring, rather than by staring at diffs.

    That's certainly true for articles not on your watchlist (or
    bad edits that
    were forgotten and are still the latest version).

    >   - Finding when flagged revisions are out of date (there
    may be a new
    >   high-trust version later)

    Well as I said I'd love to see flagged revisions and your
    system combined (in
    a way described by my previous mail). An automated system
    probably always has
    some weaknesses some clever people can abuse but it is very
    fast, while a
    hand crafted system depends on the speed of individual persons
    but is much
    harder to fool.

    > BTW, as the method is language-independent, we look forward
    to doing the
    > same for wikipedias in other languages.

    Good to know. :-)

    Arnomane

    _______________________________________________
    Wikiquality-l mailing list
    Wikiquality-l@lists.wikimedia.org
    <mailto:Wikiquality-l@lists.wikimedia.org>
    http://lists.wikimedia.org/mailman/listinfo/wikiquality-l

Re: [Wikiquality-l] Wikipedia colored according to trust

http://www.xbox.com/en-US/hardware/wheretobuy/

Akash