One thing that stood out for me in the small sample of articles I examined was the flagging of innocuous changes by casual users to correct spelling, grammar, etc. Thus a "nice-to-have" would be a "smoothing" algorithm that ignores inconsequential changes such as spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc., or the reordering of paragraphs and perhaps even sentences.) I think this would cover 90% or more of changes that are immaterial to an article's credibility.
If you want to pick out the malicious changes, you need to flag also small changes.
"Sen. Hillary Clinton did *not* vote in favor of war in Iraq"
"John Doe, born in *1947*"
The ** indicates changes.
I can very well make a system that is insensitive to small changes, but then the system would also be insensitive to many kinds of malicious tampering, and one of my goals was to make it hard for anyone to change without leaving at laest a minimal trace.
So it's a matter of goals, really.
Luca
On Dec 21, 2007 10:01 AM, Jonathan Leybovich jleybov@gmail.com wrote:
One thing that stood out for me in the small sample of articles I examined was the flagging of innocuous changes by casual users to correct spelling, grammar, etc. Thus a "nice-to-have" would be a "smoothing" algorithm that ignores inconsequential changes such as spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc., or the reordering of paragraphs and perhaps even sentences.) I think this would cover 90% or more of changes that are immaterial to an article's credibility.
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Right. Also, we need to be clear what we want this to do. It will never be great at determining fact-checked material. What it is good at is spotting the more dubious stuff, like possible vandalism. This makes the possibility of having "most trusted" stable version as discussed earlier. Small changes not only can be big in meaning, but they still attest to the trust.
If I read a sentence to change some minor thing, I still read it. If a wrongly says "he identifies himself as bisexual" or "born in 1885" rather than 1985 in a page when I edit, I am going to revert if I catch it. Even if just making some grammar/syntax cleanup. So each time people look at stuff if still attest to the page a little bit, from a vandalism perspective.
The algorithms can be made more strict to catch more general dubious info better, but it is not that bad at that already, and the stricter it gets, the more it gets under inclusive as to what is considered unlikely to be vandalized.
-Aaron Schulz
Date: Fri, 21 Dec 2007 10:34:47 -0800 From: luca@dealfaro.org To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
If you want to pick out the malicious changes, you need to flag also small changes.
"Sen. Hillary Clinton did *not* vote in favor of war in Iraq"
"John Doe, born in *1947*"
The ** indicates changes.
I can very well make a system that is insensitive to small changes, but then the system would also be insensitive to many kinds of malicious tampering, and one of my goals was to make it hard for anyone to change without leaving at laest a minimal trace.
So it's a matter of goals, really.
Luca
On Dec 21, 2007 10:01 AM, Jonathan Leybovich jleybov@gmail.com wrote:
One thing that stood out for me in the small sample of articles I examined was the flagging of innocuous changes by casual users to correct spelling, grammar, etc. Thus a "nice-to-have" would be a "smoothing" algorithm that ignores inconsequential changes such as
spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc., or the reordering of paragraphs and perhaps even sentences.) I think this
would cover 90% or more of changes that are immaterial to an article's credibility.
_______________________________________________ Wikiquality-l mailing list
Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
_________________________________________________________________ Get the power of Windows + Web with the new Windows Live. http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007
It is wise to make a note about the fact that such systems make it possible to deduce earlier in the mean that someone is a vandal or not, but it can't replace a good reader that responds to an error. This creates the rather annoying situation where a response from a casual reader should be weighted more than non-beginners, but this makes the system suceptible to users wanting to skew its metrics on specific users.
John E
Aaron Schulz skrev:
Right. Also, we need to be clear what we want this to do. It will never be great at determining fact-checked material. What it is good at is spotting the more dubious stuff, like possible vandalism. This makes the possibility of having "most trusted" stable version as discussed earlier. Small changes not only can be big in meaning, but they still attest to the trust.
If I read a sentence to change some minor thing, I still read it. If a wrongly says "he identifies himself as bisexual" or "born in 1885" rather than 1985 in a page when I edit, I am going to revert if I catch it. Even if just making some grammar/syntax cleanup. So each time people look at stuff if still attest to the page a little bit, from a vandalism perspective.
The algorithms can be made more strict to catch more general dubious info better, but it is not that bad at that already, and the stricter it gets, the more it gets under inclusive as to what is considered unlikely to be vandalized.
-Aaron Schulz
------------------------------------------------------------------------ Date: Fri, 21 Dec 2007 10:34:47 -0800 From: luca@dealfaro.org To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust If you want to pick out the malicious changes, you need to flag also small changes. "Sen. Hillary Clinton did *not* vote in favor of war in Iraq" "John Doe, born in *1947*" The ** indicates changes. I can very well make a system that is insensitive to small changes, but then the system would also be insensitive to many kinds of malicious tampering, and one of my goals was to make it hard for anyone to change without leaving at laest a minimal trace. So it's a matter of goals, really. Luca On Dec 21, 2007 10:01 AM, Jonathan Leybovich <jleybov@gmail.com <mailto:jleybov@gmail.com>> wrote: One thing that stood out for me in the small sample of articles I examined was the flagging of innocuous changes by casual users to correct spelling, grammar, etc. Thus a "nice-to-have" would be a "smoothing" algorithm that ignores inconsequential changes such as spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc., or the reordering of paragraphs and perhaps even sentences.) I think this would cover 90% or more of changes that are immaterial to an article's credibility. _______________________________________________ Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Get the power of Windows + Web with the new Windows Live. Get it now!
http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
I am not sure I am replying to the correct point, but, the system weighs an author feedback as a function of the reputation of the author. Reputation is "linear" in the sense that new feedback is simply added to the reputation. A user of reputation r gives weight log(1 + r) to hiers feedback. We use this logarithmic scaling to prevent long-time editors from forming a clique that is essentially impervious to feedback from the rest of the community (will this kind of comments get me skinned? :-)
Luca
On Dec 21, 2007 11:41 AM, John Erling Blad john.erling.blad@jeb.no wrote:
It is wise to make a note about the fact that such systems make it possible to deduce earlier in the mean that someone is a vandal or not, but it can't replace a good reader that responds to an error. This creates the rather annoying situation where a response from a casual reader should be weighted more than non-beginners, but this makes the system suceptible to users wanting to skew its metrics on specific users.
John E
Aaron Schulz skrev:
Right. Also, we need to be clear what we want this to do. It will never be great at determining fact-checked material. What it is good at is spotting the more dubious stuff, like possible vandalism. This makes the possibility of having "most trusted" stable version as discussed earlier. Small changes not only can be big in meaning, but they still attest to the trust.
If I read a sentence to change some minor thing, I still read it. If a wrongly says "he identifies himself as bisexual" or "born in 1885" rather than 1985 in a page when I edit, I am going to revert if I catch it. Even if just making some grammar/syntax cleanup. So each time people look at stuff if still attest to the page a little bit, from a vandalism perspective.
The algorithms can be made more strict to catch more general dubious info better, but it is not that bad at that already, and the stricter it gets, the more it gets under inclusive as to what is considered unlikely to be vandalized.
-Aaron Schulz
Date: Fri, 21 Dec 2007 10:34:47 -0800 From: luca@dealfaro.org To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust If you want to pick out the malicious changes, you need to flag also small changes. "Sen. Hillary Clinton did *not* vote in favor of war in Iraq" "John Doe, born in *1947*" The ** indicates changes. I can very well make a system that is insensitive to small changes, but then the system would also be insensitive to many kinds of malicious tampering, and one of my goals was to make it hard for anyone to change without leaving at laest a minimal trace. So it's a matter of goals, really. Luca On Dec 21, 2007 10:01 AM, Jonathan Leybovich <jleybov@gmail.com <mailto:jleybov@gmail.com>> wrote: One thing that stood out for me in the small sample of articlesI
examined was the flagging of innocuous changes by casual usersto
correct spelling, grammar, etc. Thus a "nice-to-have" would bea
"smoothing" algorithm that ignores inconsequential changes such as spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc., or the reordering of paragraphs and perhaps even sentences.) I think this would cover 90% or more of changes that are immaterial to an article's credibility. _______________________________________________ Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Get the power of Windows + Web with the new Windows Live. Get it now!
http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
I think that kind of weighting is correct as long as it is symmetrical (ie. do-undo -pairs weights approx the same). Clique-building is interesting although when you are able to analyze which ones interacts. Then you can punish people when their friends do bad edits. Its not as bad as it sounds, the reasoning is that often someone writes the article under guidance of some other. You don't want to punish only one user in such situations but both. John E
Luca de Alfaro skrev:
I am not sure I am replying to the correct point, but, the system weighs an author feedback as a function of the reputation of the author. Reputation is "linear" in the sense that new feedback is simply added to the reputation. A user of reputation r gives weight log(1 + r) to hiers feedback. We use this logarithmic scaling to prevent long-time editors from forming a clique that is essentially impervious to feedback from the rest of the community (will this kind of comments get me skinned? :-)
Luca
On Dec 21, 2007 11:41 AM, John Erling Blad <john.erling.blad@jeb.no mailto:john.erling.blad@jeb.no> wrote:
It is wise to make a note about the fact that such systems make it possible to deduce earlier in the mean that someone is a vandal or not, but it can't replace a good reader that responds to an error. This creates the rather annoying situation where a response from a casual reader should be weighted more than non-beginners, but this makes the system suceptible to users wanting to skew its metrics on specific users. John E Aaron Schulz skrev: > Right. Also, we need to be clear what we want this to do. It will > never be great at determining fact-checked material. What it is good > at is spotting the more dubious stuff, like possible vandalism. This > makes the possibility of having "most trusted" stable version as > discussed earlier. Small changes not only can be big in meaning, but > they still attest to the trust. > > If I read a sentence to change some minor thing, I still read it. If a > wrongly says "he identifies himself as bisexual" or "born in 1885" > rather than 1985 in a page when I edit, I am going to revert if I > catch it. Even if just making some grammar/syntax cleanup. So each > time people look at stuff if still attest to the page a little bit, > from a vandalism perspective. > > The algorithms can be made more strict to catch more general dubious > info better, but it is not that bad at that already, and the stricter > it gets, the more it gets under inclusive as to what is considered > unlikely to be vandalized. > > -Aaron Schulz > > ------------------------------------------------------------------------ > Date: Fri, 21 Dec 2007 10:34:47 -0800 > From: luca@dealfaro.org <mailto:luca@dealfaro.org> > To: wikiquality-l@lists.wikimedia.org <mailto:wikiquality-l@lists.wikimedia.org> > Subject: Re: [Wikiquality-l] Wikipedia colored according to trust > > If you want to pick out the malicious changes, you need to flag > also small changes. > > "Sen. Hillary Clinton did *not* vote in favor of war in Iraq" > > "John Doe, born in *1947*" > > The ** indicates changes. > > I can very well make a system that is insensitive to small > changes, but then the system would also be insensitive to many > kinds of malicious tampering, and one of my goals was to make it > hard for anyone to change without leaving at laest a minimal trace. > > So it's a matter of goals, really. > > Luca > > On Dec 21, 2007 10:01 AM, Jonathan Leybovich <jleybov@gmail.com <mailto:jleybov@gmail.com> > <mailto: jleybov@gmail.com <mailto:jleybov@gmail.com>>> wrote: > > One thing that stood out for me in the small sample of articles I > examined was the flagging of innocuous changes by casual users to > correct spelling, grammar, etc. Thus a "nice-to-have" would be a > "smoothing" algorithm that ignores inconsequential changes > such as > spelling corrections, etc. or the reordering of > semantically-contained > units of text (for example, reordering the line items in a > list w/o > changing the content of any particular line item, etc., or the > reordering of paragraphs and perhaps even sentences.) I think > this > would cover 90% or more of changes that are immaterial to an > article's > credibility. > > _______________________________________________ > Wikiquality-l mailing list > Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> > <mailto:Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org>> > http://lists.wikimedia.org/mailman/listinfo/wikiquality-l > > > > ------------------------------------------------------------------------ > Get the power of Windows + Web with the new Windows Live. Get it now! > <http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007> > ------------------------------------------------------------------------ > > _______________________________________________ > Wikiquality-l mailing list > Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> > http://lists.wikimedia.org/mailman/listinfo/wikiquality-l > _______________________________________________ Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> http://lists.wikimedia.org/mailman/listinfo/wikiquality-l <http://lists.wikimedia.org/mailman/listinfo/wikiquality-l>
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
How do you tell if users interact? Cross user talk page edits? If you go by edits or edits to talk, what is the difference between people in a clique and people that disagree all the time from the program's standpoint.
For identifying non-vandalized versions, cliques don't seem to pose much of a threat. The only way for that to be a problem is for people to edit stack, similar to the issue of users not using preview and making a ton of edits in a row. This could game the system sometimes, but it is also very obvious behavior when the same 2-3 new (otherwise thy would have probably been blocked by now) people pile on 8 edits after each other. Not only will it stand out, but all it would do is influence the "most trusted"/"likely unvandalized" stable version or the trust for the current revision in their favor...until someone just reverts it anyway. It's too much work for too little plus likely getting caught in the process.
The system will not be bullet proof. Admins have gone rouge before. Even for FlaggedRevs, "editor"/"surveyor" status may be abused by a very small number of people. By default it checks for email, a userpage, 150 edits spread out well, account age. Still, some people could get through and then troll. The thing is though, that it is just not worth, as they would have the rights removed and possibly be blocked immediately. And after what? Sighting some vandalism, which would just get reverted and fixed in a few minutes. To do it again would require another IP and another account back at square one again. That's just not worth it for a troll. We get vandals from people testing and joking around as well as people that just whip new accounts out their ass every minutes because it is so easy. FlaggedRevs makes it not worth it. Likewise, trying to game Article Trust does seem to very "worth it" much either.
Think of how long login didn't have captchas for such a large site. That's because nobody cared to sit there guessing around or writing bots then. If things are too hard to game, people won't care. The "rewards" of gaming this system is even way less than hacking an account. It's a bit better with FlaggedRevs because you commit to having done reviews, so reviewing vandalism goes straight against you. But still, tweaking junk edits 15 times or having the same two editors cluster 8 tiny edits after each other accomplishes little, is noticeable, and IMO, not really worth it. Also, we don't have to use AT or FR exclusively, and some combination of both (quality > sighted > autotrust > current) could avoid the multiple edit gaming issue altogether.
-Aaron Schulz
Date: Fri, 21 Dec 2007 21:49:30 +0100 From: john.erling.blad@jeb.no To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
I think that kind of weighting is correct as long as it is symmetrical (ie. do-undo -pairs weights approx the same). Clique-building is interesting although when you are able to analyze which ones interacts. Then you can punish people when their friends do bad edits. Its not as bad as it sounds, the reasoning is that often someone writes the article under guidance of some other. You don't want to punish only one user in such situations but both. John E
Luca de Alfaro skrev:
I am not sure I am replying to the correct point, but, the system weighs an author feedback as a function of the reputation of the author. Reputation is "linear" in the sense that new feedback is simply added to the reputation. A user of reputation r gives weight log(1 + r) to hiers feedback. We use this logarithmic scaling to prevent long-time editors from forming a clique that is essentially impervious to feedback from the rest of the community (will this kind of comments get me skinned? :-)
Luca
On Dec 21, 2007 11:41 AM, John Erling Blad <john.erling.blad@jeb.no mailto:john.erling.blad@jeb.no> wrote:
It is wise to make a note about the fact that such systems make it possible to deduce earlier in the mean that someone is a vandal or not, but it can't replace a good reader that responds to an error. This creates the rather annoying situation where a response from a casual reader should be weighted more than non-beginners, but this makes the system suceptible to users wanting to skew its metrics on specific users. John E Aaron Schulz skrev: > Right. Also, we need to be clear what we want this to do. It will > never be great at determining fact-checked material. What it is good > at is spotting the more dubious stuff, like possible vandalism. This > makes the possibility of having "most trusted" stable version as > discussed earlier. Small changes not only can be big in meaning, but > they still attest to the trust. > > If I read a sentence to change some minor thing, I still read it. If a > wrongly says "he identifies himself as bisexual" or "born in 1885" > rather than 1985 in a page when I edit, I am going to revert if I > catch it. Even if just making some grammar/syntax cleanup. So each > time people look at stuff if still attest to the page a little bit, > from a vandalism perspective. > > The algorithms can be made more strict to catch more general dubious > info better, but it is not that bad at that already, and the stricter > it gets, the more it gets under inclusive as to what is considered > unlikely to be vandalized. > > -Aaron Schulz > > ------------------------------------------------------------------------ > Date: Fri, 21 Dec 2007 10:34:47 -0800 > From: luca@dealfaro.org <mailto:luca@dealfaro.org> > To: wikiquality-l@lists.wikimedia.org <mailto:wikiquality-l@lists.wikimedia.org> > Subject: Re: [Wikiquality-l] Wikipedia colored according to trust > > If you want to pick out the malicious changes, you need to flag > also small changes. > > "Sen. Hillary Clinton did *not* vote in favor of war in Iraq" > > "John Doe, born in *1947*" > > The ** indicates changes. > > I can very well make a system that is insensitive to small > changes, but then the system would also be insensitive to many > kinds of malicious tampering, and one of my goals was to make it > hard for anyone to change without leaving at laest a minimal trace. > > So it's a matter of goals, really. > > Luca > > On Dec 21, 2007 10:01 AM, Jonathan Leybovich <jleybov@gmail.com <mailto:jleybov@gmail.com> > <mailto: jleybov@gmail.com <mailto:jleybov@gmail.com>>> wrote: > > One thing that stood out for me in the small sample of articles I > examined was the flagging of innocuous changes by casual users to > correct spelling, grammar, etc. Thus a "nice-to-have" would be a > "smoothing" algorithm that ignores inconsequential changes > such as > spelling corrections, etc. or the reordering of > semantically-contained > units of text (for example, reordering the line items in a > list w/o > changing the content of any particular line item, etc., or the > reordering of paragraphs and perhaps even sentences.) I think > this > would cover 90% or more of changes that are immaterial to an > article's > credibility. > > _______________________________________________ > Wikiquality-l mailing list > Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> > <mailto:Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org>> > http://lists.wikimedia.org/mailman/listinfo/wikiquality-l > > > > ------------------------------------------------------------------------ > Get the power of Windows + Web with the new Windows Live. Get it now! > <http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007> > ------------------------------------------------------------------------ > > _______________________________________________ > Wikiquality-l mailing list > Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> > http://lists.wikimedia.org/mailman/listinfo/wikiquality-l > _______________________________________________ Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> http://lists.wikimedia.org/mailman/listinfo/wikiquality-l <http://lists.wikimedia.org/mailman/listinfo/wikiquality-l>
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
_________________________________________________________________ i’m is proud to present Cause Effect, a series about real people making a difference. http://im.live.com/Messenger/IM/MTV/?source=text_Cause_Effect
Note that I don't say clique detection should be implemented, I say this is a very interesting option to get running if possible. There are basically two possible solution, or rather I'm not aware of any other that is possible to implement. One is to use identified cooperation between two or more persons to be able to edit an otherwise locked article, the other is to identify cooperation by checking how two or more users interact. If two persons starting to interact in a positive way at any one article it is highly unlikely the goes to war on another article. War between users can be readily detected by simple statistical means (aka automatic classification of statements).
Now the second problem. In trust-accumulating systems you don't have to pile up edits in a single article to game the system. You use two bots. One bad-bot introduces some identifiable error in many articles, and then another good-bot fix those errors. The good-bot will quickly gain high reputation. Even so if the bad-bot changes ip-address often so it goes undetected. This has to be solved somehow, and to say it involves to much work to do something like that is plainly a wrong solution to the problem. I think it is solvable, and I think at least one solution is the fact that all such systems will produce highly unusual usage metrics. It is then mostly a matter of detecting the unusual metrics and sound an alarm bell. If it is possible to incorporate "physical means" that makes it impossible to game the system it is even better. For example, a radio communication system can be jammed, but using frequency hopping radios makes them susceptible to jamming. If they are hopping fast enough they stays ahead of the jammer. If they are hopping faster than the radio waves can reach the jammer then it is physically impossible to use smart jammers.
Don't create a system that can be easilly fooled by more or less smart bots. Make it physically impossible to fool the system with a bot.
1) decrease credits when time between edits goes down 2) never give more credits to revert an edit then given when making the original edit 3) weight the edits according to the amount of contribution, yet different styles of contributions should give the same net result in credits 4) decrease credits when cooperation are detected and the reason behind the cooperation can't be detected 5) increase credits when there are a positive cooperation
A slight modification of 2 is to release it if there goes sufficient time.
John E Blad
Aaron Schulz skrev:
How do you tell if users interact? Cross user talk page edits? If you go by edits or edits to talk, what is the difference between people in a clique and people that disagree all the time from the program's standpoint.
For identifying non-vandalized versions, cliques don't seem to pose much of a threat. The only way for that to be a problem is for people to edit stack, similar to the issue of users not using preview and making a ton of edits in a row. This could game the system sometimes, but it is also very obvious behavior when the same 2-3 new (otherwise thy would have probably been blocked by now) people pile on 8 edits after each other. Not only will it stand out, but all it would do is influence the "most trusted"/"likely unvandalized" stable version or the trust for the current revision in their favor...until someone just reverts it anyway. It's too much work for too little plus likely getting caught in the process.
The system will not be bullet proof. Admins have gone rouge before. Even for FlaggedRevs, "editor"/"surveyor" status may be abused by a very small number of people. By default it checks for email, a userpage, 150 edits spread out well, account age. Still, some people could get through and then troll. The thing is though, that it is just not worth, as they would have the rights removed and possibly be blocked immediately. And after what? Sighting some vandalism, which would just get reverted and fixed in a few minutes. To do it again would require another IP and another account back at square one again. That's just not worth it for a troll. We get vandals from people testing and joking around as well as people that just whip new accounts out their ass every minutes because it is so easy. FlaggedRevs makes it not worth it. Likewise, trying to game Article Trust does seem to very "worth it" much either.
Think of how long login didn't have captchas for such a large site. That's because nobody cared to sit there guessing around or writing bots then. If things are too hard to game, people won't care. The "rewards" of gaming this system is even way less than hacking an account. It's a bit better with FlaggedRevs because you commit to having done reviews, so reviewing vandalism goes straight against you. But still, tweaking junk edits 15 times or having the same two editors cluster 8 tiny edits after each other accomplishes little, is noticeable, and IMO, not really worth it. Also, we don't have to use AT or FR exclusively, and some combination of both (quality > sighted
autotrust > current) could avoid the multiple edit gaming issue
altogether.
-Aaron Schulz
Date: Fri, 21 Dec 2007 21:49:30 +0100 From: john.erling.blad@jeb.no To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
I think that kind of weighting is correct as long as it is symmetrical (ie. do-undo -pairs weights approx the same). Clique-building is interesting although when you are able to analyze which ones interacts. Then you can punish people when their friends do bad edits. Its not as bad as it sounds, the reasoning is that often someone writes the article under guidance of some other. You don't want to punish only one user in such situations but both. John E
Luca de Alfaro skrev:
I am not sure I am replying to the correct point, but, the system weighs an author feedback as a function of the reputation of the
author.
Reputation is "linear" in the sense that new feedback is simply added to the reputation. A user of reputation r gives weight log(1 + r) to hiers feedback. We use this logarithmic scaling to prevent long-time editors from forming a clique that is essentially impervious to feedback from the rest of the community (will this kind of comments get me skinned? :-)
Luca
On Dec 21, 2007 11:41 AM, John Erling Blad <john.erling.blad@jeb.no mailto:john.erling.blad@jeb.no> wrote:
It is wise to make a note about the fact that such systems make it possible to deduce earlier in the mean that someone is a vandal or not, but it can't replace a good reader that responds to an error. This creates the rather annoying situation where a response from a casual reader should be weighted more than non-beginners, but this makes the system suceptible to users wanting to skew its metrics on specific users.
John E
Aaron Schulz skrev:
Right. Also, we need to be clear what we want this to do. It will never be great at determining fact-checked material. What it is good at is spotting the more dubious stuff, like possible vandalism.
This
makes the possibility of having "most trusted" stable version as discussed earlier. Small changes not only can be big in meaning, but they still attest to the trust.
If I read a sentence to change some minor thing, I still read
it. If a
wrongly says "he identifies himself as bisexual" or "born in 1885" rather than 1985 in a page when I edit, I am going to revert if I catch it. Even if just making some grammar/syntax cleanup. So each time people look at stuff if still attest to the page a little bit, from a vandalism perspective.
The algorithms can be made more strict to catch more general dubious info better, but it is not that bad at that already, and the
stricter
it gets, the more it gets under inclusive as to what is considered unlikely to be vandalized.
-Aaron Schulz
Date: Fri, 21 Dec 2007 10:34:47 -0800 From: luca@dealfaro.org mailto:luca@dealfaro.org To: wikiquality-l@lists.wikimedia.org
mailto:wikiquality-l@lists.wikimedia.org
Subject: Re: [Wikiquality-l] Wikipedia colored according to
trust
If you want to pick out the malicious changes, you need to flag also small changes.
"Sen. Hillary Clinton did *not* vote in favor of war in Iraq"
"John Doe, born in *1947*"
The ** indicates changes.
I can very well make a system that is insensitive to small changes, but then the system would also be insensitive to many kinds of malicious tampering, and one of my goals was to make it hard for anyone to change without leaving at laest a minimal
trace.
So it's a matter of goals, really.
Luca
On Dec 21, 2007 10:01 AM, Jonathan Leybovich
<jleybov@gmail.com mailto:jleybov@gmail.com
<mailto: jleybov@gmail.com mailto:jleybov@gmail.com>> wrote:
One thing that stood out for me in the small sample of
articles I
examined was the flagging of innocuous changes by casual
users to
correct spelling, grammar, etc. Thus a "nice-to-have"
would be a
"smoothing" algorithm that ignores inconsequential changes such as spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc.,
or the
reordering of paragraphs and perhaps even sentences.) I
think
this would cover 90% or more of changes that are immaterial
to an
article's credibility.
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org
<mailto:Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org>
Get the power of Windows + Web with the new Windows Live. Get it
now!
http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org mailto:Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
i’m is proud to present Cause Effect, a series about real people making a difference. Learn more
http://im.live.com/Messenger/IM/MTV/?source=text_Cause_Effect
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
That would still be quite a feat. The bot would have to IP hop over several A-classes. Make some random vandalism that looks like random vandalism (rather than the same set of garbage or random characters/blanking). The IP would have to change enough for it to not look like one bot on RC. The good bot would have to give a time pause before reverting to let other beat it to the punch. Also, if all it did was revert bad bot, then as I said, the bad bot's IP would really have to be all over the place, unless the bot made random accounts too. It would still take a while to build up trust this way...and even with the max, it will still take several edits to get bad content white. This would have to be good enough to set the "most trusted" version. And even if this is done on some pages, it will get reverted and the user blocked, and they have to use another "good bot".
It certainly is always better to be less easily spoofed, and if there are good practical things that can stop this without making the software too underinclusive, I'm all for it. I said earlier that users in the formal bot group should not count as adding credibility. In this vein, we could do the opposite and have groups that increase the max trust (credits) a user can have. That way the default max trust can be lowered to deal further with stuff like good/bad bot networks. This would work if integrated with FlaggedRevs, as the 'editor' group could have a higher max trust limit.
-Aaron Schulz
Date: Sat, 22 Dec 2007 00:42:27 +0100 From: john.erling.blad@jeb.no To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
Note that I don't say clique detection should be implemented, I say this is a very interesting option to get running if possible. There are basically two possible solution, or rather I'm not aware of any other that is possible to implement. One is to use identified cooperation between two or more persons to be able to edit an otherwise locked article, the other is to identify cooperation by checking how two or more users interact. If two persons starting to interact in a positive way at any one article it is highly unlikely the goes to war on another article. War between users can be readily detected by simple statistical means (aka automatic classification of statements).
Now the second problem. In trust-accumulating systems you don't have to pile up edits in a single article to game the system. You use two bots. One bad-bot introduces some identifiable error in many articles, and then another good-bot fix those errors. The good-bot will quickly gain high reputation. Even so if the bad-bot changes ip-address often so it goes undetected. This has to be solved somehow, and to say it involves to much work to do something like that is plainly a wrong solution to the problem. I think it is solvable, and I think at least one solution is the fact that all such systems will produce highly unusual usage metrics. It is then mostly a matter of detecting the unusual metrics and sound an alarm bell. If it is possible to incorporate "physical means" that makes it impossible to game the system it is even better. For example, a radio communication system can be jammed, but using frequency hopping radios makes them susceptible to jamming. If they are hopping fast enough they stays ahead of the jammer. If they are hopping faster than the radio waves can reach the jammer then it is physically impossible to use smart jammers.
Don't create a system that can be easilly fooled by more or less smart bots. Make it physically impossible to fool the system with a bot.
- decrease credits when time between edits goes down
- never give more credits to revert an edit then given when making the
original edit 3) weight the edits according to the amount of contribution, yet different styles of contributions should give the same net result in credits 4) decrease credits when cooperation are detected and the reason behind the cooperation can't be detected 5) increase credits when there are a positive cooperation
A slight modification of 2 is to release it if there goes sufficient time.
John E Blad
Aaron Schulz skrev:
How do you tell if users interact? Cross user talk page edits? If you go by edits or edits to talk, what is the difference between people in a clique and people that disagree all the time from the program's standpoint.
For identifying non-vandalized versions, cliques don't seem to pose much of a threat. The only way for that to be a problem is for people to edit stack, similar to the issue of users not using preview and making a ton of edits in a row. This could game the system sometimes, but it is also very obvious behavior when the same 2-3 new (otherwise thy would have probably been blocked by now) people pile on 8 edits after each other. Not only will it stand out, but all it would do is influence the "most trusted"/"likely unvandalized" stable version or the trust for the current revision in their favor...until someone just reverts it anyway. It's too much work for too little plus likely getting caught in the process.
The system will not be bullet proof. Admins have gone rouge before. Even for FlaggedRevs, "editor"/"surveyor" status may be abused by a very small number of people. By default it checks for email, a userpage, 150 edits spread out well, account age. Still, some people could get through and then troll. The thing is though, that it is just not worth, as they would have the rights removed and possibly be blocked immediately. And after what? Sighting some vandalism, which would just get reverted and fixed in a few minutes. To do it again would require another IP and another account back at square one again. That's just not worth it for a troll. We get vandals from people testing and joking around as well as people that just whip new accounts out their ass every minutes because it is so easy. FlaggedRevs makes it not worth it. Likewise, trying to game Article Trust does seem to very "worth it" much either.
Think of how long login didn't have captchas for such a large site. That's because nobody cared to sit there guessing around or writing bots then. If things are too hard to game, people won't care. The "rewards" of gaming this system is even way less than hacking an account. It's a bit better with FlaggedRevs because you commit to having done reviews, so reviewing vandalism goes straight against you. But still, tweaking junk edits 15 times or having the same two editors cluster 8 tiny edits after each other accomplishes little, is noticeable, and IMO, not really worth it. Also, we don't have to use AT or FR exclusively, and some combination of both (quality > sighted
autotrust > current) could avoid the multiple edit gaming issue
altogether.
-Aaron Schulz
Date: Fri, 21 Dec 2007 21:49:30 +0100 From: john.erling.blad@jeb.no To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
I think that kind of weighting is correct as long as it is symmetrical (ie. do-undo -pairs weights approx the same). Clique-building is interesting although when you are able to analyze which ones interacts. Then you can punish people when their friends do bad edits. Its not as bad as it sounds, the reasoning is that often someone writes the article under guidance of some other. You don't want to punish only one user in such situations but both. John E
Luca de Alfaro skrev:
I am not sure I am replying to the correct point, but, the system weighs an author feedback as a function of the reputation of the
author.
Reputation is "linear" in the sense that new feedback is simply added to the reputation. A user of reputation r gives weight log(1 + r) to hiers feedback. We use this logarithmic scaling to prevent long-time editors from forming a clique that is essentially impervious to feedback from the rest of the community (will this kind of comments get me skinned? :-)
Luca
On Dec 21, 2007 11:41 AM, John Erling Blad <john.erling.blad@jeb.no mailto:john.erling.blad@jeb.no> wrote:
It is wise to make a note about the fact that such systems make it possible to deduce earlier in the mean that someone is a vandal or not, but it can't replace a good reader that responds to an error. This creates the rather annoying situation where a response from a casual reader should be weighted more than non-beginners, but this makes the system suceptible to users wanting to skew its metrics on specific users.
John E
Aaron Schulz skrev:
Right. Also, we need to be clear what we want this to do. It will never be great at determining fact-checked material. What it is good at is spotting the more dubious stuff, like possible vandalism.
This
makes the possibility of having "most trusted" stable version as discussed earlier. Small changes not only can be big in meaning, but they still attest to the trust.
If I read a sentence to change some minor thing, I still read
it. If a
wrongly says "he identifies himself as bisexual" or "born in 1885" rather than 1985 in a page when I edit, I am going to revert if I catch it. Even if just making some grammar/syntax cleanup. So each time people look at stuff if still attest to the page a little bit, from a vandalism perspective.
The algorithms can be made more strict to catch more general dubious info better, but it is not that bad at that already, and the
stricter
it gets, the more it gets under inclusive as to what is considered unlikely to be vandalized.
-Aaron Schulz
Date: Fri, 21 Dec 2007 10:34:47 -0800 From: luca@dealfaro.org mailto:luca@dealfaro.org To: wikiquality-l@lists.wikimedia.org
mailto:wikiquality-l@lists.wikimedia.org
Subject: Re: [Wikiquality-l] Wikipedia colored according to
trust
If you want to pick out the malicious changes, you need to flag also small changes.
"Sen. Hillary Clinton did *not* vote in favor of war in Iraq"
"John Doe, born in *1947*"
The ** indicates changes.
I can very well make a system that is insensitive to small changes, but then the system would also be insensitive to many kinds of malicious tampering, and one of my goals was to make it hard for anyone to change without leaving at laest a minimal
trace.
So it's a matter of goals, really.
Luca
On Dec 21, 2007 10:01 AM, Jonathan Leybovich
<jleybov@gmail.com mailto:jleybov@gmail.com
<mailto: jleybov@gmail.com mailto:jleybov@gmail.com>> wrote:
One thing that stood out for me in the small sample of
articles I
examined was the flagging of innocuous changes by casual
users to
correct spelling, grammar, etc. Thus a "nice-to-have"
would be a
"smoothing" algorithm that ignores inconsequential changes such as spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc.,
or the
reordering of paragraphs and perhaps even sentences.) I
think
this would cover 90% or more of changes that are immaterial
to an
article's credibility.
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org
<mailto:Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org>
Get the power of Windows + Web with the new Windows Live. Get it
now!
http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org mailto:Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
i’m is proud to present Cause Effect, a series about real people making a difference. Learn more
http://im.live.com/Messenger/IM/MTV/?source=text_Cause_Effect
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
_________________________________________________________________ Share life as it happens with the new Windows Live. http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_122007
Actually, account creation has a captcha, so it wouldn't be using accounts likely.
-Aaron Schulz
From: jschulz_4587@msn.com To: wikiquality-l@lists.wikimedia.org Date: Fri, 21 Dec 2007 19:55:04 -0500 Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
That would still be quite a feat. The bot would have to IP hop over several A-classes. Make some random vandalism that looks like random vandalism (rather than the same set of garbage or random characters/blanking). The IP would have to change enough for it to not look like one bot on RC. The good bot would have to give a time pause before reverting to let other beat it to the punch. Also, if all it did was revert bad bot, then as I said, the bad bot's IP would really have to be all over the place, unless the bot made random accounts too. It would still take a while to build up trust this way...and even with the max, it will still take several edits to get bad content white. This would have to be good enough to set the "most trusted" version. And even if this is done on some pages, it will get reverted and the user blocked, and they have to use another "good bot".
It certainly is always better to be less easily spoofed, and if there are good practical things that can stop this without making the software too underinclusive, I'm all for it. I said earlier that users in the formal bot group should not count as adding credibility. In this vein, we could do the opposite and have groups that increase the max trust (credits) a user can have. That way the default max trust can be lowered to deal further with stuff like good/bad bot networks. This would work if integrated with FlaggedRevs, as the 'editor' group could have a higher max trust limit.
-Aaron Schulz
_________________________________________________________________ Get the power of Windows + Web with the new Windows Live. http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007
There are public algorithms with methods to defeat captchas. I haven't looked into the version used by Wikipedia, but compared to some other versions that are cracked it seems likely it isn't safe. (Actually the basic technique used by the version in Wikipedia has some serious design flaws that makes it possible to crack it by brute force, not that I find it very likely that anyone will try this.)
John E
Aaron Schulz skrev:
Actually, account creation has a captcha, so it wouldn't be using accounts likely.
-Aaron Schulz
------------------------------------------------------------------------ From: jschulz_4587@msn.com To: wikiquality-l@lists.wikimedia.org Date: Fri, 21 Dec 2007 19:55:04 -0500 Subject: Re: [Wikiquality-l] Wikipedia colored according to trust That would still be quite a feat. The bot would have to IP hop over several A-classes. Make some random vandalism that looks like random vandalism (rather than the same set of garbage or random characters/blanking). The IP would have to change enough for it to not look like one bot on RC. The good bot would have to give a time pause before reverting to let other beat it to the punch. Also, if all it did was revert bad bot, then as I said, the bad bot's IP would really have to be all over the place, unless the bot made random accounts too. It would still take a while to build up trust this way...and even with the max, it will still take several edits to get bad content white. This would have to be good enough to set the "most trusted" version. And even if this is done on some pages, it will get reverted and the user blocked, and they have to use another "good bot". It certainly is always better to be less easily spoofed, and if there are good practical things that can stop this without making the software too underinclusive, I'm all for it. I said earlier that users in the formal bot group should not count as adding credibility. In this vein, we could do the opposite and have groups that increase the max trust (credits) a user can have. That way the default max trust can be lowered to deal further with stuff like good/bad bot networks. This would work if integrated with FlaggedRevs, as the 'editor' group could have a higher max trust limit. -Aaron Schulz
Get the power of Windows + Web with the new Windows Live. Get it now!
http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Sorry for bringing this age old thread up again, but reading through it again, I often thought: things would be easier, if Metadata were not in the articles themselves, but separate. Then, the version history would be trimmed dramatically and nobody could build up trust (meaning both the real and the algorithmic sense) by category-edits only. This would make life dramatically easier for the bots and easier for users.
Is getting the metadata (category, interwikis, Personendaten on de) out of the articles remotely feasible? Is the positive impact as good as I think?
Best
Philipp
I am sure that, if it were of interest, one coudl remove interwiki links and categories while analizing the pages... or am I missing something? I however would hesitate to remove the information from the revision history: how to revert damage to the meta-information? (Yes, better ways to compress the revision history are very much needed, for the working version, but this is another issue)... I hope my comments are on topic; I am not sure I understand the point.
Luca
On Jan 30, 2008 1:27 PM, P. Birken pbirken@gmail.com wrote:
Sorry for bringing this age old thread up again, but reading through it again, I often thought: things would be easier, if Metadata were not in the articles themselves, but separate. Then, the version history would be trimmed dramatically and nobody could build up trust (meaning both the real and the algorithmic sense) by category-edits only. This would make life dramatically easier for the bots and easier for users.
Is getting the metadata (category, interwikis, Personendaten on de) out of the articles remotely feasible? Is the positive impact as good as I think?
Best
Philipp
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Currently, metadata is simply in the article body. However, in theory, metadata could be stored and edited in a separate place with its own revision history, like the discussion is separate from the article. I hope the idea is now more clear?
Philipp
2008/1/30, Luca de Alfaro luca@dealfaro.org:
I am sure that, if it were of interest, one coudl remove interwiki links and categories while analizing the pages... or am I missing something? I however would hesitate to remove the information from the revision history: how to revert damage to the meta-information? (Yes, better ways to compress the revision history are very much needed, for the working version, but this is another issue)... I hope my comments are on topic; I am not sure I understand the point.
Luca
On Jan 30, 2008 1:27 PM, P. Birken pbirken@gmail.com wrote:
Sorry for bringing this age old thread up again, but reading through it again, I often thought: things would be easier, if Metadata were not in the articles themselves, but separate. Then, the version history would be trimmed dramatically and nobody could build up trust (meaning both the real and the algorithmic sense) by category-edits only. This would make life dramatically easier for the bots and easier for users.
Is getting the metadata (category, interwikis, Personendaten on de) out of the articles remotely feasible? Is the positive impact as good as I think?
Best
Philipp
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
I think separating the metadata makes sense because it cleans up the markup language, and makes the metadata itself more easily extensible. But, I do not think that the fact that the metadata is embedded in the pages makes our trust/reputation computation more complex -- it would be fairly easy to separate out the metadata at analysis stage, if we wanted to do that. On the negative side, separating the metadata requires new DB tables, etc etc, making also the format of the dumps more complex (my guess). In balance, though, it seems a good idea, but I am speaking here without knowing the facts.
Luca
On Jan 30, 2008 1:41 PM, P. Birken pbirken@gmail.com wrote:
Currently, metadata is simply in the article body. However, in theory, metadata could be stored and edited in a separate place with its own revision history, like the discussion is separate from the article. I hope the idea is now more clear?
Philipp
2008/1/30, Luca de Alfaro luca@dealfaro.org:
I am sure that, if it were of interest, one coudl remove interwiki links
and
categories while analizing the pages... or am I missing something? I however would hesitate to remove the information from the revision history: how to revert damage to the meta-information? (Yes, better ways to compress the revision history are very much
needed,
for the working version, but this is another issue)... I hope my comments are on topic; I am not sure I understand the point.
Luca
On Jan 30, 2008 1:27 PM, P. Birken pbirken@gmail.com wrote:
Sorry for bringing this age old thread up again, but reading through it again, I often thought: things would be easier, if Metadata were not in the articles themselves, but separate. Then, the version history would be trimmed dramatically and nobody could build up trust (meaning both the real and the algorithmic sense) by category-edits only. This would make life dramatically easier for the bots and easier for users.
Is getting the metadata (category, interwikis, Personendaten on de) out of the articles remotely feasible? Is the positive impact as good as I think?
Best
Philipp
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
"The bot would have to IP hop over several A-classes", this is whats called a tor-network.
One very problematic type of vandalism that usually goes undetected are introduction of nonexisting iw-links. Another are wrong categories. The list of possible options are fairly long.
Note that I don't say this isn't a good tool, I say that it should be carefully inspected for ways to game the system, and if possible, adding features that makes it impossible to use those flaws.
John E Blad
Aaron Schulz skrev:
That would still be quite a feat. The bot would have to IP hop over several A-classes. Make some random vandalism that looks like random vandalism (rather than the same set of garbage or random characters/blanking). The IP would have to change enough for it to not look like one bot on RC. The good bot would have to give a time pause before reverting to let other beat it to the punch. Also, if all it did was revert bad bot, then as I said, the bad bot's IP would really have to be all over the place, unless the bot made random accounts too. It would still take a while to build up trust this way...and even with the max, it will still take several edits to get bad content white. This would have to be good enough to set the "most trusted" version. And even if this is done on some pages, it will get reverted and the user blocked, and they have to use another "good bot".
It certainly is always better to be less easily spoofed, and if there are good practical things that can stop this without making the software too underinclusive, I'm all for it. I said earlier that users in the formal bot group should not count as adding credibility. In this vein, we could do the opposite and have groups that increase the max trust (credits) a user can have. That way the default max trust can be lowered to deal further with stuff like good/bad bot networks. This would work if integrated with FlaggedRevs, as the 'editor' group could have a higher max trust limit.
-Aaron Schulz
Date: Sat, 22 Dec 2007 00:42:27 +0100 From: john.erling.blad@jeb.no To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
Note that I don't say clique detection should be implemented, I say this is a very interesting option to get running if possible. There are basically two possible solution, or rather I'm not aware of any other that is possible to implement. One is to use identified cooperation between two or more persons to be able to edit an otherwise locked article, the other is to identify cooperation by checking how two or more users interact. If two persons starting to interact in a positive way at any one article it is highly unlikely the goes to war on another article. War between users can be readily detected by simple statistical means (aka automatic classification of statements).
Now the second problem. In trust-accumulating systems you don't have to pile up edits in a single article to game the system. You use two bots. One bad-bot introduces some identifiable error in many articles, and then another good-bot fix those errors. The good-bot will quickly gain high reputation. Even so if the bad-bot changes ip-address often so it goes undetected. This has to be solved somehow, and to say it involves to much work to do something like that is plainly a wrong solution to the problem. I think it is solvable, and I think at least one solution is the fact that all such systems will produce highly unusual usage metrics. It is then mostly a matter of detecting the unusual metrics and sound an alarm bell. If it is possible to incorporate "physical means" that makes it impossible to game the system it is even better. For example, a radio communication system can be jammed, but using frequency hopping radios makes them susceptible to jamming. If they are hopping fast enough they stays ahead of the jammer. If they are hopping faster than the radio waves can reach the jammer then it is physically impossible to use smart jammers.
Don't create a system that can be easilly fooled by more or less smart bots. Make it physically impossible to fool the system with a bot.
- decrease credits when time between edits goes down
- never give more credits to revert an edit then given when making the
original edit 3) weight the edits according to the amount of contribution, yet different styles of contributions should give the same net result in
credits
- decrease credits when cooperation are detected and the reason behind
the cooperation can't be detected 5) increase credits when there are a positive cooperation
A slight modification of 2 is to release it if there goes sufficient
time.
John E Blad
Aaron Schulz skrev:
How do you tell if users interact? Cross user talk page edits? If you go by edits or edits to talk, what is the difference between people in a clique and people that disagree all the time from the program's standpoint.
For identifying non-vandalized versions, cliques don't seem to pose much of a threat. The only way for that to be a problem is for people to edit stack, similar to the issue of users not using preview and making a ton of edits in a row. This could game the system sometimes, but it is also very obvious behavior when the same 2-3 new (otherwise thy would have probably been blocked by now) people pile on 8 edits after each other. Not only will it stand out, but all it would do is influence the "most trusted"/"likely unvandalized" stable version or the trust for the current revision in their favor...until someone just reverts it anyway. It's too much work for too little plus likely getting caught in the process.
The system will not be bullet proof. Admins have gone rouge before. Even for FlaggedRevs, "editor"/"surveyor" status may be abused by a very small number of people. By default it checks for email, a userpage, 150 edits spread out well, account age. Still, some people could get through and then troll. The thing is though, that it is just not worth, as they would have the rights removed and possibly be blocked immediately. And after what? Sighting some vandalism, which would just get reverted and fixed in a few minutes. To do it again would require another IP and another account back at square one again. That's just not worth it for a troll. We get vandals from people testing and joking around as well as people that just whip new accounts out their ass every minutes because it is so easy. FlaggedRevs makes it not worth it. Likewise, trying to game Article Trust does seem to very "worth it" much either.
Think of how long login didn't have captchas for such a large site. That's because nobody cared to sit there guessing around or writing bots then. If things are too hard to game, people won't care. The "rewards" of gaming this system is even way less than hacking an account. It's a bit better with FlaggedRevs because you commit to having done reviews, so reviewing vandalism goes straight against you. But still, tweaking junk edits 15 times or having the same two editors cluster 8 tiny edits after each other accomplishes little, is noticeable, and IMO, not really worth it. Also, we don't have to use AT or FR exclusively, and some combination of both (quality > sighted
autotrust > current) could avoid the multiple edit gaming issue
altogether.
-Aaron Schulz
Date: Fri, 21 Dec 2007 21:49:30 +0100 From: john.erling.blad@jeb.no To: wikiquality-l@lists.wikimedia.org Subject: Re: [Wikiquality-l] Wikipedia colored according to trust
I think that kind of weighting is correct as long as it is
symmetrical
(ie. do-undo -pairs weights approx the same). Clique-building is interesting although when you are able to analyze which ones
interacts.
Then you can punish people when their friends do bad edits. Its
not as
bad as it sounds, the reasoning is that often someone writes the
article
under guidance of some other. You don't want to punish only one
user in
such situations but both. John E
Luca de Alfaro skrev:
I am not sure I am replying to the correct point, but, the system weighs an author feedback as a function of the reputation of the
author.
Reputation is "linear" in the sense that new feedback is
simply added
to the reputation. A user of reputation r gives weight log(1 + r) to hiers feedback. We use this logarithmic scaling to prevent long-time editors from forming a clique that is essentially impervious to feedback
from the
rest of the community (will this kind of comments get me
skinned? :-)
Luca
On Dec 21, 2007 11:41 AM, John Erling Blad
<john.erling.blad@jeb.no
mailto:john.erling.blad@jeb.no> wrote:
It is wise to make a note about the fact that such systems make it possible to deduce earlier in the mean that someone is a vandal or not, but it can't replace a good reader that responds to an error. This creates the rather annoying situation where a response from a
casual
reader should be weighted more than non-beginners, but this
makes the
system suceptible to users wanting to skew its metrics on specific users.
John E
Aaron Schulz skrev:
Right. Also, we need to be clear what we want this to do. It
will
never be great at determining fact-checked material. What it
is good
at is spotting the more dubious stuff, like possible vandalism.
This
makes the possibility of having "most trusted" stable version as discussed earlier. Small changes not only can be big in
meaning, but
they still attest to the trust.
If I read a sentence to change some minor thing, I still read
it. If a
wrongly says "he identifies himself as bisexual" or "born in
1885"
rather than 1985 in a page when I edit, I am going to revert
if I
catch it. Even if just making some grammar/syntax cleanup.
So each
time people look at stuff if still attest to the page a
little bit,
from a vandalism perspective.
The algorithms can be made more strict to catch more general
dubious
info better, but it is not that bad at that already, and the
stricter
it gets, the more it gets under inclusive as to what is
considered
unlikely to be vandalized.
-Aaron Schulz
Date: Fri, 21 Dec 2007 10:34:47 -0800 From: luca@dealfaro.org mailto:luca@dealfaro.org To: wikiquality-l@lists.wikimedia.org
mailto:wikiquality-l@lists.wikimedia.org
Subject: Re: [Wikiquality-l] Wikipedia colored according to
trust
If you want to pick out the malicious changes, you need to flag also small changes.
"Sen. Hillary Clinton did *not* vote in favor of war in Iraq"
"John Doe, born in *1947*"
The ** indicates changes.
I can very well make a system that is insensitive to small changes, but then the system would also be insensitive to many kinds of malicious tampering, and one of my goals was to make it hard for anyone to change without leaving at laest a minimal
trace.
So it's a matter of goals, really.
Luca
On Dec 21, 2007 10:01 AM, Jonathan Leybovich
<jleybov@gmail.com mailto:jleybov@gmail.com
<mailto: jleybov@gmail.com mailto:jleybov@gmail.com>> wrote:
One thing that stood out for me in the small sample of
articles I
examined was the flagging of innocuous changes by casual
users to
correct spelling, grammar, etc. Thus a "nice-to-have"
would be a
"smoothing" algorithm that ignores inconsequential changes such as spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc.,
or the
reordering of paragraphs and perhaps even sentences.) I
think
this would cover 90% or more of changes that are immaterial
to an
article's credibility.
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org
<mailto:Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org>
Get the power of Windows + Web with the new Windows Live. Get it
now!
http://www.windowslive.com?ocid=TXT_TAGHM_Wave2_powerofwindows_122007
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org
mailto:Wikiquality-l@lists.wikimedia.org
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org mailto:Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
i’m is proud to present Cause Effect, a series about real people making a difference. Learn more http://im.live.com/Messenger/IM/MTV/?source=text_Cause_Effect
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Share life as it happens with the new Windows Live. Share now! http://www.windowslive.com/share.html?ocid=TXT_TAGHM_Wave2_sharelife_122007
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
I think you misunderstand, its not about not flagging small changes. Its about to high learning rate on small changes. When it is to high you can simply run a spell checking bot for a few days to become an "expert". You don't want this. You want someone who contribute a lot of text without being reverted to become an expert. You probably also want to analyse the form of the changes so a lot of small spell checkings is weighted even less than correcting a year. Ie., changing from something that is possibly an misspelling into something right is weighted less than changing something correctly spelled into something else that is also correctly spelled. John E Blad
Luca de Alfaro skrev:
If you want to pick out the malicious changes, you need to flag also small changes.
"Sen. Hillary Clinton did *not* vote in favor of war in Iraq"
"John Doe, born in *1947*"
The ** indicates changes.
I can very well make a system that is insensitive to small changes, but then the system would also be insensitive to many kinds of malicious tampering, and one of my goals was to make it hard for anyone to change without leaving at laest a minimal trace.
So it's a matter of goals, really.
Luca
On Dec 21, 2007 10:01 AM, Jonathan Leybovich <jleybov@gmail.com mailto:jleybov@gmail.com> wrote:
One thing that stood out for me in the small sample of articles I examined was the flagging of innocuous changes by casual users to correct spelling, grammar, etc. Thus a "nice-to-have" would be a "smoothing" algorithm that ignores inconsequential changes such as spelling corrections, etc. or the reordering of semantically-contained units of text (for example, reordering the line items in a list w/o changing the content of any particular line item, etc., or the reordering of paragraphs and perhaps even sentences.) I think this would cover 90% or more of changes that are immaterial to an article's credibility. _______________________________________________ Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org <mailto:Wikiquality-l@lists.wikimedia.org> http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Wikiquality-l mailing list Wikiquality-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikiquality-l
Am Freitag, 21. Dezember 2007 20:31:18 schrieb John Erling Blad:
I think you misunderstand, its not about not flagging small changes. Its about to high learning rate on small changes. When it is to high you can simply run a spell checking bot for a few days to become an "expert". You don't want this.
This has already been noted by Luca himself: The current algorithms has two weaknesses (better say optimisation potential as it is already quite good at this unfinished stage): * Sock puppets pushing "trust" * Splitting of contributions into small edits
See http://lists.wikimedia.org/pipermail/wikiquality-l/2007-December/000405.html for details (also about possible fixes).
Furthermore as noted by Aaron and by me a combination of automated (trust color code) and hand crafted (stable versions) system maybe will be able to overcome the weaknesses of both concepts, see: http://lists.wikimedia.org/pipermail/wikiquality-l/2007-December/000393.html http://lists.wikimedia.org/pipermail/wikiquality-l/2007-December/000394.html
Arnomane
wikiquality-l@lists.wikimedia.org