Hi folks,
So glad to see the old and new ML teams have an open discussion about this subject.
I understand that the team might prefer to have several tickets for different issues, but the discussion about the general approach to the different models is of interest to many people and is more easily digested on email. I would suggest to continue discussing the merits of the current strategy (and not necessarily of a model or another) on email.
* One model per wiki or overall
This is a tough one. :) As a user, I remember how hard it was for Romanian speakers to complete the training data for damaging/goodfaith and would prefer to not have to do it again.
However, I'm also worried that some specificities of larger wikis would creep in the output, leading to reverts that would normally not happen on my wiki. For instance, smaller settlements are not accepted on enwp, while they are accepted on rowp. I don't know how to test it myself, and I haven't seen anything about it in the research.
Another problem I have is I'm not sure how the revert-risk score should be matched against custom damaging/goodfaith thresholds. Ate there some guidelines on this except "test"?
* Multiple criteria VS a single score
I think the discussion has been very much about reverts, but as Sj said, each of these scores are a slightly different facet. Is there data available on the prevalence of other use-cases or is everyone just writing revert bots?
On the long run, I believe an unique model good enough can be developed for revert bots. However, it would be great if there were some clear quality criteria that the community can verify and the old models are maintained for a wiki until we are sure the new model passes that criteria on that wiki.
A change in hosting should not be the guiding force in any team's roadmap, but the needs of its users.
Have a good weekend,
Strainu
Pe sâmbătă, 23 septembrie 2023, Luca Toscano <ltoscano@wikimedia.org> a scris:
>
>
> On Fri, Sep 22, 2023 at 11:34 PM Aaron Halfaker <aaron.halfaker@gmail.com> wrote:
>>
>> All fine points. As you can see, I've filed some phab tasks where I saw a clear opportunity to do so.
>
> Thanks a lot! We are going to review them next week and decide the next steps, but we'd like to proceed anyway to migrate ores to ores-legacy on Monday (this will allow us to free some old nodes that need to be decommed etc..). Adding features later on to the models on Lift Wing should be doable, and our goal is to transition away from ores-legacy in a few months (to avoid maintaining too many systems). The timeline is not yet set in stone, we'll update this mailing list when the time comes (and we'll follow up with the remaining users of ores-legacy as well). To summarize: we start with Ores -> Ores Legacy on Monday, and we'll do Ores Legacy -> Lift Wing in a second step.
>>
>> > as mentioned before all the models that currently run on ORES are available in both ores-legacy and Lift Wing.
>>
>> I thought I read that damaging and goodfaith models are going to be replaced. Should I instead read that they are likely to remain available for the foreseeable future? When I asked about a community discussion about the transition from damaging/goodfaith to revertrisk, I was imagining that many people who use those predictions might have an opinion about them going away. E.g. people who use the relevant filters in RecentChanges. Maybe I missed the discussions about that.
>
> This is a good point, I'll clarify the documentation on Wikitech. Until models are used we'll not remove them from Lift Wing, but we'll propose to use Revert Risk where it is suited since it is a model family on which we decided to invest time and efforts. Basic maintenance will be performed on the goodfaith/damaging/articlequality/etc.. models on Lift Wing, but we don't have (at the moment) any bandwidth to guarantee retraining or more complex workflows on them. This is why we used the term "deprecated" on Wikitech, but we need to specify what we mean to avoid confusion. Thanks for the feedback :)
>
>>
>> I haven't seen a mention of the article quality or article topic models in the docs. Are those also going to remain available? I have some user scripts that use these models and are relatively widely used. I didn't notice anyone reaching out. ... So I checked and setting a User-Agent on my user scripts doesn't actually change the User-Agent. I've read that you need to set "Api-User-Agent" instead, but that causes a CORS error when querying ORES. I'll file a bug.
>
> Will update the docs as well, as mentioned above we'll keep the current ORES models available on Lift Wing. Eventually new models will be proposed by Research and other teams (like Revert Risk), and at that point we (as ML team) will decide what recommendation to give. Nothing will be removed from Lift Wing if there are active users on it, but we'll certainly try to reduce the amount of models to maintain (based on common functionality etc..), so some changes will be proposed in the future.
> Luca