FWIW, all of the ORES false positives that I've
seen so far have been
anonymous users fixing single words, for example, correcting verb tense or
changing to a more specific word. ORES typically marks these as damaging
with a high confidence regardless of the substance of the change.
On Wed, Aug 24, 2016 at 6:07 AM, Amir Ladsgroup <ladsgroup(a)gmail.com>
wrote:
I also want to add that you can change ores
sensitivity in your
preferences
and add "We deliberately set the default threshold so low to capture all
vandalism cases so false positives are expected unlike anti-vandalism bot
that set the threshold so high to capture only vandalism cases (and don't
have false positives)."
Best
On Wed, Aug 24, 2016 at 2:13 AM Aaron Halfaker <aaron.halfaker(a)gmail.com>
wrote:
Thanks Luis! :)
And I just finished setting up a new labeling campaign for English
Wikipedia. This data will help us train/test more accurate models.
See
https://en.wikipedia.org/wiki/Wikipedia:Labels/Edit_quality for
instructions on how to get started.
-Aaron
On Tue, Aug 23, 2016 at 4:05 PM, Luis Villa <luis(a)lu.is> wrote:
> Thanks for the detailed explanation, Aaron. As always your work is a
> model in transparency for the rest of us :)
>
>
> On Tue, Aug 23, 2016 at 12:40 PM Aaron Halfaker <
aaron.halfaker(a)gmail.com>
> wrote:
>
>> Hi Luis! Thanks for taking a look.
>>
>> First, I should say that false-positives should be expected. We're
>> working on better signaling in the UI so that you can differentiate
the
>> edits that ORES is confident about and
those that it isn't confident
about
>> -- but are still worth your review.
>>
>> So, in order to avoid a bias feedback loop, we don't want to feed any
>> observations you made *using* ORES back into the model -- since ORES'
>> prediction itself could bias your assessment and we'd re-perpetuate
that
>> bias. Still, we can use these
misclassification reports to direct our
>> attention to problematic behaviors in the model. We use the Wiki
Labels
>> system[1] to gather reviews of random
samples of edits from
Wikipedians in
>> order to train the model.
>>
>> *Misclassification reports:*
>> See
>>
https://meta.wikimedia.org/wiki/Research:Revision_scoring_
as_a_service/Misclassifications/Edit_quality
>>
>> We're still working out the Right(TM) way to report false positives.
>> Right now, we ask that you do so on-wiki and in the future, we'll be
>> exploring a nicer interface so that you can report them while using
the
>> tool. We review these misclassification
reports manually to focus
our work
>> on the models and to report progress
made. This data is never
directly
>> used in training the machine learning
models due to issues around
bias.
>>
>> *Wiki labels campaigns:*
>> In order to avoid the biases in who gets reviewed and why, we generate
>> random samples of edits for review using our Wiki Labels[1] system.
We've
>> completed a labeling campaign for English
Wikipedia[2], but we could
run an
>> additional campaign to gather more data.
I'll get that set up and
respond
>> to this message when it is ready.
>>
>> 1.
https://meta.wikimedia.org/wiki/Wiki_labels
>> 2.
https://en.wikipedia.org/wiki/Wikipedia:Labels/Edit_quality
>>
>> -Aaron
>>
>> On Tue, Aug 23, 2016 at 1:30 PM, Luis Villa <luis(a)lu.is> wrote:
>>
>>> Very cool! Is there any way for users of this tool to help train it?
>>> For example, the first four things it flagged in my watchlist were
all
>>> false positives (next 5-6 were
correctly flagged.) It'd be nice to
be able
>>
to contribute to training the model somehow when we see these
>> false-positives.
>>
>> On Tue, Aug 23, 2016 at 11:10 AM Amir Ladsgroup <ladsgroup(a)gmail.com
>>> wrote:
>>>
>>>> We (The Revision Scoring Team
>>>> <https://meta.wikimedia.org/wiki/Research:Revision_scoring_
as_a_service#Team>)
>>>> are happy to announce the
deployment of the ORES
>>>> <https://meta.wikimedia.org/wiki/ORES> review tool
>>>> <https://www.mediawiki.org/wiki/ORES_review_tool> as a beta
feature
>>>> <https://en.wikipedia.org/wiki/Special:Preferences#mw-prefse
ction-betafeatures>
>>>> on *English Wikipedia*. Once
enabled, ORES highlights edits that
are
>>>> likely to be damaging in
Special:RecentChanges
>>>> <https://en.wikipedia.org/wiki/Special:RecentChanges>,
>>>> Special:Watchlist
<https://en.wikipedia.org/wiki/Special:Watchlist>
>>>> and Special:Contributions
>>>> <https://en.wikipedia.org/wiki/Special:Contributions> to help you
>>>> prioritize your patrolling work. ORES detects damaging edits using a
>>>> basic prediction model based on past damage
>>>> <https://meta.wikimedia.org/wiki/Research:Automated_classifi
cation_of_edit_quality>.
>>>> ORES is an experimental
technology. We encourage you to take
advantage of
>>>> it but also to be skeptical of
the predictions made. It's a tool to
support
>>>> you – it can't replace you.
Please reach out to us with your
questions and
>>>> concerns.
>>>> Documentationmw:ORES review tool
>>>> <https://www.mediawiki.org/wiki/ORES_review_tool>,
mw:Extension:ORES
>>>>
<https://www.mediawiki.org/wiki/Extension:ORES>, and m:ORES
>>>> <https://meta.wikimedia.org/wiki/ORES>Bugs & feature requests
>>>>
https://phabricator.wikimedia.org/tag/revision-scoring-as-a-
service-backlog/
>>> IRC#wikimedia-aiconnect
>>> <http://webchat.freenode.net/?channels=#wikimedia-ai>
>>> Sincerely,Amir from the Revision Scoring team
>>> _______________________________________________
>>> AI mailing list
>>> AI(a)lists.wikimedia.org
>>>
https://lists.wikimedia.org/mailman/listinfo/ai
>>>
>>
>> _______________________________________________
>> AI mailing list
>> AI(a)lists.wikimedia.org
>>
https://lists.wikimedia.org/mailman/listinfo/ai
>>
>>
> _______________________________________________
> AI mailing list
> AI(a)lists.wikimedia.org
>
https://lists.wikimedia.org/mailman/listinfo/ai
>
_______________________________________________
AI mailing list
AI(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/ai
_______________________________________________
AI mailing list
AI(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/ai
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l