[Foundation-l] Frequency of Seeing Bad Versions - now with traffic data

[Gregory Maxwell]

On Thu, Aug 27, 2009 at 8:24 PM, Thomas Dalton<thomas.dalton at gmail.com> wrote:
> 2009/8/28 Anthony <wikimail at inbox.org>:
>> On Thu, Aug 27, 2009 at 7:58 PM, Stephen Bain <stephen.bain at gmail.com>wrote:
>>> On Fri, Aug 28, 2009 at 4:58 AM, Anthony<wikimail at inbox.org> wrote:
>>> > It seems to me to be begging the question.  You don't answer the question
>>> > "how bad is vandalism" by assuming that vandalism is generally reverted.
>>> Can you suggest a better metric then?
>> I must admit I don't understand the question.
>
> He means what would you measure in order to draw conclusions about the
> severity of vandalism.

The obvious methodology would be to take a large random sample and
hand classify it. It's not rocket science.

By having multiple people perform the classification you could measure
the confidence of the classification.

This is somewhat labor intensive, but only somewhat as it doesn't take
an inordinate number of samples to produce representative results.
This should be the gold standard for this kind of measurement as it
would be much closer to what people actually want to know than most
machine metrics.

If the results of this kind of study have good agreement with
mechanical proxy metrics (such as machine detected vandalism) our
confidence in those proxies will increase, if they disagree it will
provide an opportunity to improve the proxies.

These are techniques widely used in other fields.