Re: [Wikitech-l] [AI] Deployment of ORES review tool in Englis Wikipedia as a beta feature

Hi Luis! Thanks for taking a look. First, I should say that false-positives should be expected. We're working on better signaling in the UI so that you can differentiate the edits that ORES is confident about and those that it isn't confident about -- but are still worth your review. So, in order to avoid a bias feedback loop, we don't want to feed any observations you made *using* ORES back into the model -- since ORES' prediction itself could bias your assessment and we'd re-perpetuate that bias. Still, we can use these misclassification reports to direct our attention to problematic behaviors in the model. We use the Wiki Labels system[1] to gather reviews of random samples of edits from Wikipedians in order to train the model. *Misclassification reports:* See https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Misc… We're still working out the Right(TM) way to report false positives. Right now, we ask that you do so on-wiki and in the future, we'll be exploring a nicer interface so that you can report them while using the tool. We review these misclassification reports manually to focus our work on the models and to report progress made. This data is never directly used in training the machine learning models due to issues around bias. *Wiki labels campaigns:* In order to avoid the biases in who gets reviewed and why, we generate random samples of edits for review using our Wiki Labels[1] system. We've completed a labeling campaign for English Wikipedia[2], but we could run an additional campaign to gather more data. I'll get that set up and respond to this message when it is ready. 1. https://meta.wikimedia.org/wiki/Wiki_labels 2. https://en.wikipedia.org/wiki/Wikipedia:Labels/Edit_quality -Aaron On Tue, Aug 23, 2016 at 1:30 PM, Luis Villa <luis(a)lu.is> wrote: _______________________________________________ AI mailing list AI(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ai