Hi Chris & ML team,
Good to see LiftWing is finally becoming a reality. There are a few things in the documentation that I would like to clarify.
1. In [1], the bot owner is encouraged to move to the
revertrisk score. However, in [2], it's explicitly mentioned that the model should not be used for "Auto-removing edits that a user makes without another editor in the loop". So, should bot owners currently reverting based on goodfaith and damaging scores explore the new models? If so, do you have any suggestions on how to automatically match thresholds between the old and new models?
Diego (from the Research team) answered this bit afaics, but I read it in another Wikitech-l thread (maybe it is my email reader, not sure, but I wanted to point it out in case you missed it).
Quoting Diego:
"""
Sorry for the confusion, we have updated this model card. You can use this model for "automatically reverting content" as you were using ORES. Here you can see the model's performance comparison.
Our current recommendation is to use the Language Agnostic model for this task (patrolling bots). The Multilingual model is performing better for IP Edits, but we are still working on improving its stability. Within the next 3 months we expect to improve Language Agnostic accuracy in anonymous edits, and also Multilingual model stability.
"""
2. I could not find any reference regarding the ores scores exposed through other APIs (specifically the RC API [3]). Will those be available going forward? Under which names?
I am very ignorant about RC APIs, but if you want to explore this part more please open a task in Phabricator with the Machine-Learning-team tag, we'll try to research what is possible and get back to you. We'd be also curious to know the use case, to figure out how to best support it.
3. Will it still be possible to (re-)train existing and new model for a specific wiki? How and when?
So far the ML team concentrated all the efforts in the serving infrastructure (Lift Wing), meanwhile the training part is still to be decided. In [1] we added info about how to request to host a model on Lift Wing, but we didn't provide any automated way to train or retrain the models over time. It is a big effort that we'll tackle in the future, we'll keep this list updated as much as possible. All the models that we host now have been trained on big nodes like the Analytics statistics ones [2], but every re-train is manual and ad-hoc for a specific use case. We are also strongly encouraging people to migrate away from Revscoring models (goodfaith, damaging, etc..) as much as possible, we'd prefer not to to retrain those (where possible) and migrate people to more modern solutions (like Revert Risk). Having said this, if you have any specific request please open a Phabricator task with the Machine-Learning-team tag and we'll evaluate the use case.
Thanks!
Luca