It seems that what Ward and others are getting at is that it would be
useful to have precision and recall measures for Luca's trust metric.
Of course, the metric can't possibly know it when a brand new user
contributes unusually high quality text to the encyclopedia.
Nonetheless, it seems that a tool such as Amazon's Mechanical Turk
could allow us to easily measure how often false positives and false
negatives occur using random sampling. Although your hammer was not designed for their nail, I imagine it would do quite well.<br>
<br><br><div class="gmail_quote">On Dec 20, 2007 9:04 AM, Luca de Alfaro &lt;<a href="mailto:luca@soe.ucsc.edu">luca@soe.ucsc.edu</a>&gt; wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
You are evaluating the coloring against a performance criterion that is not the one we designed it for.<br><br>Our coloring gives orange color to new information that has been added by low-reputation authors.&nbsp; New information by high-reputation authors is light orange.&nbsp; As the information is revised, it gains trust. 
<br><br>Thus, our coloring answers the question, intuitively: has this information been revised already?&nbsp; Have reputable authors looked at it? <br>&nbsp;<br>You are asking the question: how much information colored orange is questionable?
<br>This is a different question, and we will never be able to do well, for the simple reason that it is well known that a lot of the correct factual information on Wikipedia comes from occasional contributors, including anonymous authors, and those occasional contributors and anonymous will have low reputation in most conceivable reputation systems. 
<br><br>We do not plan to do any large-scale human study.&nbsp; For one, we don&#39;t have the resources.&nbsp; For another, in the very limited tests we did, the notion of &quot;questionable&quot; was so subjective that our data contained a HUGE amount of noise.&nbsp; We asked to rank edits as -1 (bad), 0 (neutral), +1 (good).&nbsp; The probability that two of us agreed was somewhere below 60%.&nbsp; We decided this was not a good way to go. 
<br><br>The results of our data-driven evaluation on a random sample of 1000 articles with at least 200 revisions each showed that (quoting from our paper): <br><ul><li>Recall of deletions. We consider the recall of low-trust as a predictor for deletions. We show that text in the lowest 50% of trust values constitutes only 
3.4% of the text of articles, yet corresponds to 66% of the&nbsp; text that is deleted from one revision to the next.</li></ul><ul><li>Precision of deletions.&nbsp; We consider the precision of low-trust as a predictor for deletions. We show that text that is in the bottom half of trust values has a probability of 33% of being deleted in the very next revision, in&nbsp; contrast with the 
1.9% probability for general text.&nbsp; The deletion probability raises to 62% for text in the bottom 20%&nbsp; of trust values.<br><br></li><li>Trust of average vs. deleted text. We consider the trust distribution
of all text, compared to the trust distribution to the text that is
deleted.&nbsp; We show that 90% of the text overall had trust at least 76%,
while the average trust for deleted text was 33%.</li></ul><ul><li>Trust as a predictor of lifespan.&nbsp; We select words uniformly at random, and we consider the statistical correlation between the trust of the word at the moment of sampling, and the future lifespan of the word.&nbsp; We show that words with the highest trust have an expected future lifespan that is 
4.5 times longer than words with no trust.&nbsp; We remark that this is a proper test, since the trust at the time of sampling depends only on the history of the word prior to sampling.<br></li></ul><font color="#888888">Luca
</font><div><div></div><div class="Wj3C7c"><br><br><div class="gmail_quote">
On Dec 20, 2007 6:12 AM, Desilets, Alain &lt;<a href="mailto:Alain.Desilets@nrc-cnrc.gc.ca" target="_blank">Alain.Desilets@nrc-cnrc.gc.ca</a>&gt; wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">










<div link="blue" vlink="purple" lang="EN-CA">

<div>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">Here is my feedback based on looking at a few pages on topics
that I know very well.</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">Agile Software Development</span></p>

<p style="text-indent: -18pt;"><span style="font-size: 11pt; font-family: Symbol; color: rgb(31, 73, 125);"><span>ˇ<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span style="font-size: 11pt; color: rgb(31, 73, 125);"><a href="http://wiki-trust.cse.ucsc.edu/index.php/Agile_software_development" target="_blank">http://wiki-trust.cse.ucsc.edu/index.php/Agile_software_development
</a></span></p>

<p style="text-indent: -18pt;"><span style="font-size: 11pt; font-family: Symbol; color: rgb(31, 73, 125);"><span>ˇ<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span style="font-size: 11pt; color: rgb(31, 73, 125);">Not bad. I counted 13 highlighted items, 5 of which I would say
are questionable.</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">Usability</span></p>

<p style="text-indent: -18pt;"><span style="font-size: 11pt; font-family: Symbol; color: rgb(31, 73, 125);"><span>ˇ<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span style="font-size: 11pt; color: rgb(31, 73, 125);"><a href="http://wiki-trust.cse.ucsc.edu/index.php/Usability" target="_blank">http://wiki-trust.cse.ucsc.edu/index.php/Usability</a></span></p>

<p style="text-indent: -18pt;"><span style="font-size: 11pt; font-family: Symbol; color: rgb(31, 73, 125);"><span>ˇ<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span style="font-size: 11pt; color: rgb(31, 73, 125);">Not as good. 14 highlighted items 3 of which I would say are
questionable.</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">Open Source Software</span></p>

<p style="text-indent: -18pt;"><span style="font-size: 11pt; font-family: Symbol; color: rgb(31, 73, 125);"><span>ˇ<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span style="font-size: 11pt; color: rgb(31, 73, 125);"><a href="http://wiki-trust.cse.ucsc.edu/index.php/Open_source_software" target="_blank">http://wiki-trust.cse.ucsc.edu/index.php/Open_source_software
</a></span></p>

<p style="text-indent: -18pt;"><span style="font-size: 11pt; font-family: Symbol; color: rgb(31, 73, 125);"><span>ˇ<span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
</span></span></span><span style="font-size: 11pt; color: rgb(31, 73, 125);">Not so good either. 23 highlighted items, 3 of which I would say
are questionable.</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">This is a very small sample, but it&#39;s all I have time to
do. It will be interesting to see how other people rate the precision of the
highlightings on a wider set of topics. Based on these three examples, it&#39;s
not entirely clear to me that this system would help me identify questionable
items in topics that I am not so familiar with.</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">Are you planning to do a larger scale evaluation with human
judges? An issue in that kind of study is to avoid favourable or disfavourable
bias on the part of the judges. Also, you have to make sure that your algorithm
is doing better than random guessing (in other words, there may be so many questionable
phrases in a wiki page that random guessing would be bound to guess right ounce
out of every say, 5 times). One way to avoid these issues would be to produce
pages where half of the highlightings are produced by your system, and the
other half are highlighting a randomly selected contiguous contribution by a
single author.</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">I think this is really interesting work worth doing, btw. I just
don&#39;t know how useful it is in its current state.</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">Cheers,</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">Alain D</span><span style="font-size: 11pt; color: rgb(31, 73, 125);" lang="FR-CA">ésilets</span><span style="font-size: 11pt; color: rgb(31, 73, 125);"></span></p>



<div style="border-style: none none none solid; border-color: -moz-use-text-color -moz-use-text-color -moz-use-text-color blue; border-width: medium medium medium 1.5pt; padding: 0cm 0cm 0cm 4pt;">

<p><span style="font-size: 11pt; color: rgb(31, 73, 125);">&nbsp;</span></p>

</div>

</div>

</div>


</blockquote></div><br>
</div></div><br>_______________________________________________<br>Wiki-research-l mailing list<br><a href="mailto:Wiki-research-l@lists.wikimedia.org">Wiki-research-l@lists.wikimedia.org</a><br><a href="http://lists.wikimedia.org/mailman/listinfo/wiki-research-l" target="_blank">
http://lists.wikimedia.org/mailman/listinfo/wiki-research-l</a><br><br></blockquote></div><br>