The University of Santa Cruz/California has an interesting demo up that computes author trust based on whether users' edits are kept/improved or reverted. It then highlights passages of the text according to the computed reputation of the author who added them:
http://trust.cse.ucsc.edu/ http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at Wikimania.
This is described at http://meta.wikimedia.org/wiki/User:Agtfjott/Article_quality_and_user_credit...
The first work on such a system is an article at (From 12. mar 2006) http://no.wikipedia.org/wiki/Bruker:Agtfjott/Artikkelkvalitet_og_brukeretter...
This article is now deleted. Links to the article at meta are posted both here, at #wikipedia, #wikimedia and several other places so it can hardly be an unknown approach to anyone.
It is also discussed to set it up as a project together with Norwegian Computing Center http://en.wikipedia.org/wiki/Norwegian_Computing_Center
John E
Erik Moeller wrote:
The University of Santa Cruz/California has an interesting demo up that computes author trust based on whether users' edits are kept/improved or reverted. It then highlights passages of the text according to the computed reputation of the author who added them:
http://trust.cse.ucsc.edu/ http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at
Wikimania.
From a very fast reading of the paper at
http://www.soe.ucsc.edu/~luca/papers/07/wikiwww2007.pdf
I found the following problems with the paper, and some problems not covered at all * It does not take into account the problems introduced by dyslectic users. Such users can contribute a lot of info, yet the info they contribute can be completely rewritten. * User credits/reputation is unbound and therefore very difficult to use for decisions, should an user be allowed to edit a specific article or not? * Collaborations are not taken into account. This is very important as users working together usually makes very good contributions. * Users having expertise in one area, which reflects in the contributions. This will have impact on the user reputation at similar articles, ie within the same category. * Credits/Reputation is not only given as a function of an users own edits, but also as a result by the quality of the article he or she edits. The quality of the article is again given by the reputation of the users editing them, thereby creating a highly interconnected system. To anly take user contributions and reverts into account is to simple. * When credits/reputation is used for decisions like excluding an user from editing an article the decision will reinforce its own decision, so after making the decision it will become harder and harder to get over the limit to be allowed to edit an article. * If both user credits/reputation and article quality are in use there is easily introduced an instability which results in a situation whereby an user are given less credits for editing an article than the article itself are given quality points, leading to a situation where the user is blocked from editing his own article. This situation is most likely to happen when there is introduced a non-linear growth factor in credits/reputation or quality.
It could be more that I didn't notice.
John E
John Erling Blad wrote:
This is described at http://meta.wikimedia.org/wiki/User:Agtfjott/Article_quality_and_user_credit...
The first work on such a system is an article at (From 12. mar 2006) http://no.wikipedia.org/wiki/Bruker:Agtfjott/Artikkelkvalitet_og_brukeretter...
This article is now deleted. Links to the article at meta are posted both here, at #wikipedia, #wikimedia and several other places so it can hardly be an unknown approach to anyone.
It is also discussed to set it up as a project together with Norwegian Computing Center http://en.wikipedia.org/wiki/Norwegian_Computing_Center
John E
Erik Moeller wrote:
The University of Santa Cruz/California has an interesting demo up that computes author trust based on whether users' edits are kept/improved or reverted. It then highlights passages of the text according to the computed reputation of the author who added them:
http://trust.cse.ucsc.edu/ http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at
Wikimania.
I'll crosspost this to admins at no.wp
John E
John Erling Blad wrote:
This is described at http://meta.wikimedia.org/wiki/User:Agtfjott/Article_quality_and_user_credit...
The first work on such a system is an article at (From 12. mar 2006) http://no.wikipedia.org/wiki/Bruker:Agtfjott/Artikkelkvalitet_og_brukeretter...
This article is now deleted. Links to the article at meta are posted both here, at #wikipedia, #wikimedia and several other places so it can hardly be an unknown approach to anyone.
It is also discussed to set it up as a project together with Norwegian Computing Center http://en.wikipedia.org/wiki/Norwegian_Computing_Center
John E
Erik Moeller wrote:
The University of Santa Cruz/California has an interesting demo up that computes author trust based on whether users' edits are kept/improved or reverted. It then highlights passages of the text according to the computed reputation of the author who added them:
http://trust.cse.ucsc.edu/ http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at
Wikimania.
--- Erik Moeller erik@wikimedia.org wrote:
The University of Santa Cruz/California has an interesting demo up that computes author trust based on whether users' edits are kept/improved or reverted. It then highlights passages of the text according to the computed reputation of the author who added them:
http://trust.cse.ucsc.edu/ http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at Wikimania.
Wow - that is even better than the idea of user-driven trust metrics. On top of that, it encourages activity we want to encourage; creating content that lasts and is built on. It also looks like they might have a better diff algorithm than we do.
-- mav
____________________________________________________________________________________ Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center. http://autos.yahoo.com/green_center/
Thanks for the comments! Yes, we very much wanted a system that
- does not change the day-to-day Wikipedia experience (it worked so well so far, let's not change what worked; would people be put off, or strange behaviors encouraged, by user-to-user ratings?), - encourages lasting content, but does not punish people whose content is rewritten/improved: people are mainly punished for reversions or partial reversions. - does not display reputation associated with authors (newbies to the Wikipedia provide a good share of the factual content, as they include many domain experts, so it's important not to put them off) - but still gives useful information to visitors on the trust of text (and lots more can be done, e.g., getting on request the last high trust version, ...)
As you point out, getting text diff to work is not trivial, and it took us a long time to get something we liked; we had to write it from scratch... the idea is given in the WWW07 paper: a greedy algorithm, that matches longest substrings first, giving however a bias in favor of substrings that occur in the same relative position in the pages. Moreover, we keep track not only of the text present in a page (the "live" text), but also of the text that used to be present, but has been deleted (the "dead" text). If you don't do this, reverts (and partial reverts) are not dealt with correctly. We think that even better can be done, in fact (everything can always be improved), but we haven't had a chance yet.
Luca
On 7/30/07, Daniel Mayer maveric149@yahoo.com wrote:
--- Erik Moeller erik@wikimedia.org wrote:
The University of Santa Cruz/California has an interesting demo up that computes author trust based on whether users' edits are kept/improved or reverted. It then highlights passages of the text according to the computed reputation of the author who added them:
http://trust.cse.ucsc.edu/ http://enwiki-trust.cse.ucsc.edu/index.php/Special:Random
NB, this is still very experimental, but it seems promising.
Luca de Alfaro, who did most of this work, will also be presenting at
Wikimania.
Wow - that is even better than the idea of user-driven trust metrics. On top of that, it encourages activity we want to encourage; creating content that lasts and is built on. It also looks like they might have a better diff algorithm than we do.
-- mav
____________________________________________________________________________________Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center. http://autos.yahoo.com/green_center/
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
--- Luca de Alfaro luca@soe.ucsc.edu wrote:
Thanks for the comments! Yes, we very much wanted a system that
- does not change the day-to-day Wikipedia experience (it worked so
well so far, let's not change what worked; would people be put off, or strange behaviors encouraged, by user-to-user ratings?),
- encourages lasting content, but does not punish people whose content
is rewritten/improved: people are mainly punished for reversions or partial reversions.
- does not display reputation associated with authors (newbies to the
Wikipedia provide a good share of the factual content, as they include many domain experts, so it's important not to put them off)
- but still gives useful information to visitors on the trust of text
(and lots more can be done, e.g., getting on request the last high trust version, ...)
As you point out, getting text diff to work is not trivial, and it took us a long time to get something we liked; we had to write it from scratch... the idea is given in the WWW07 paper: a greedy algorithm, that matches longest substrings first, giving however a bias in favor of substrings that occur in the same relative position in the pages. Moreover, we keep track not only of the text present in a page (the "live" text), but also of the text that used to be present, but has been deleted (the "dead" text). If you don't do this, reverts (and partial reverts) are not dealt with correctly. We think that even better can be done, in fact (everything can always be improved), but we haven't had a chance yet.
Please continue your work! :) Also, will UCSC allow you to license your work under the GPL or a compatible license? It would be a shame if your work could not be incorporated into MediaWiki one day.
Oh - I'm sure you already thought of this, but please make sure this tool only works on the main namespace since that is where all the articles (content) hang out. Your analysis will be skewed if it looks at non-article content since much of which tends to be preserved as is and without modification for years.
I also dont want to reward those who do almost all their editing outside of articles by giving their edits an incorrectly higher rank. We really need to encourage editing of content, not arguing on talk and policy pages.
-- mav
____________________________________________________________________________________ Luggage? GPS? Comic books? Check out fitting gifts for grads at Yahoo! Search http://search.yahoo.com/search?fr=oni_on_mail&p=graduation+gifts&cs=...
Dear All,
We plan to release the code under the GPL (I assume this is fine for MediaWiki; otherwise, we can release it under what they wish if there is interest into an integration). We haven't done a release yet mainly because the code is in flux (we just finished implementing trust; we will now tinker with it a bit until we are satisfied). Note however, that the code is currently written to batch-process .xml dumps, where all the revisions of a page are given sequentially. The reputation we use is well-suited to become an on-line system (the amount of computation to do for every edit is small), but that would require some code modifications, since the order of presentation of the revisions would be different (every time someone edits).
The main thing that would be needed to implement this as an on-line (real-time) system in WikiMedia is to store, for each page, a little bit of extra information in some auxiliary DB table (such as trust and version of origin for each word of the last revision; reputation value for each author). The information would be much smaller than the total history size; it would be essentially proportional to the size of the last revision (so the storage requirements would not change much).
To implement this system with minimal disruption to the current Wikipedia servers, one could make sure that every time the main Mediawiki servers get an edit, they forward a copy of the edit to another server, which runs the reputation/trust stuff, and which can then serve the colored pages to interested visitors. This way, one could bring the trust/reputation system online with minimal disruption to the Wikipedia servers (another advantage of not requiring user-to-user ratings).
As for the namespaces, yes, we know, we just analyze the main name space (or at least we try -- let me know if you find some article in the demo that does not belong there).
Best, Luca
On 7/31/07, Daniel Mayer maveric149@yahoo.com wrote:
--- Luca de Alfaro luca@soe.ucsc.edu wrote:
Thanks for the comments! Yes, we very much wanted a system that
- does not change the day-to-day Wikipedia experience (it worked so
well so far, let's not change what worked; would people be put off,
or
strange behaviors encouraged, by user-to-user ratings?),
- encourages lasting content, but does not punish people whose
content
is rewritten/improved: people are mainly punished for reversions or
partial
reversions.
- does not display reputation associated with authors (newbies to the
Wikipedia provide a good share of the factual content, as they
include many
domain experts, so it's important not to put them off)
- but still gives useful information to visitors on the trust of text
(and lots more can be done, e.g., getting on request the last high trust version, ...)
As you point out, getting text diff to work is not trivial, and it took
us a
long time to get something we liked; we had to write it from scratch...
the
idea is given in the WWW07 paper: a greedy algorithm, that matches
longest
substrings first, giving however a bias in favor of substrings that
occur in
the same relative position in the pages. Moreover, we keep track not
only
of the text present in a page (the "live" text), but also of the text
that
used to be present, but has been deleted (the "dead" text). If you
don't do
this, reverts (and partial reverts) are not dealt with correctly. We think that even better can be done, in fact (everything can always be improved), but we haven't had a chance yet.
Please continue your work! :) Also, will UCSC allow you to license your work under the GPL or a compatible license? It would be a shame if your work could not be incorporated into MediaWiki one day.
Oh - I'm sure you already thought of this, but please make sure this tool only works on the main namespace since that is where all the articles (content) hang out. Your analysis will be skewed if it looks at non-article content since much of which tends to be preserved as is and without modification for years.
I also don't want to reward those who do almost all their editing outside of articles by giving their edits an incorrectly higher rank. We really need to encourage editing of content, not arguing on talk and policy pages.
-- mav
____________________________________________________________________________________Luggage? GPS? Comic books? Check out fitting gifts for grads at Yahoo! Search http://search.yahoo.com/search?fr=oni_on_mail&p=graduation+gifts&cs=...
wikiquality-l@lists.wikimedia.org