Sometimes you just want to score a few revisions in ORES and remembering the URL structure is hard. So, we've build a simple scoring user-interface
that will allow you to more easily score a set of edits.
New API version
We've been consistently getting requests to include more information in ORES' responses. In order to make space for this new information, we needed to change the structure of responses. But we wanted to do this without breaking the tools that are already using ORES. So, we've developed a versioning scheme that will allow you to take advantage of new functionality when you are ready. The same old API will continue to be available at https://ores.wmflabs.org/scores/
, but we've added two additional paths on top of this.
Feature values & injection
Breaking change -- new models
We've been experimenting with new learning algorithms to make ORES work better and we've found that we get better results with gradient boosting
and random forest
strategies than we do with the current linear svc
models. We'd like to get these new, better models deployed as soon as possible, but with the new algorithm comes a change in the range of probabilities returned by the model. So, when we deploy this change, any tools that uses hard-coded thresholds on ORES' prediction probabilities will suddenly start behaving strangely. Regretfully, we haven't found a way around this problem, so we're announcing the change now and we plan to deploy this BREAKING CHANGE on April 7th
. Please subscribe to the AI mailinglist
or watch our project page [[:m:ORES
]] to catch announcements of future changes and new functionality.
In order to make sure we don't end up in the same situation the next time we want to change an algorithm, we've included a suite of evaluation statistics with each model. The filter_rate_at_recall(0.9), filter_rate_at_recall(0.75), and recall_at_fpr(0.1) thresholds represent three critical thresholds (should review, needs review, and definitely damaging -- respectively) that can be used to automatically configure your wiki tool. You can find out these thresholds for your model of choice by adding the ?model_info parameter to requests. So, come breaking change, we strongly recommend basing your thresholds on these statistics in the future. We'll be working to submit patches to tools that use ORES in the next week to implement this flexibility. Hopefully, all you'll need to do is worth with us on those.