Hi Fastily, we are aware of the use case for matching commons pages/images/sha1s between commons/big wikis and other wikis, as it has come up many times. I'm cataloging all the comments and examples that have come up in the last 5 months in order to provide categorized input to the
parent task so that the engineering teams can think of solutions. I'll share it publicly once it is in a presentable state.
We did some exploration a while ago (from Huji's examples), you can see some notebooks with python approaches
here, but there is too much data and doing the same takes a very long time and can be impractical. If you want to give it a try have a look at the notebooks, I don't think the code is too memory intensive, specially in bd808s notebook using the API, and Raspberry Pis could maybe handle it.
It is more complex and error-prone, for sure, so disabling those reports and waiting is sadly the option right now, until a suitable solution for this is found.
So, to answer your question:
Is there going to be a replacement for this functionality?
I can't promise anything yet but I can assure you the teams involved in these systems are aware of the need for this functionality and will be looking into how to provide it to make these reports/bots/queries viable.
We will send updates or new info to the cloud lists, and you can subscribe to these tasks if you want to follow more closely:
A little late to the party, I just learned about this change today.
I maintain a number of
bot tasks and
database reports on enwp that rely on cross-wiki joins (mostly page title joins between enwp and Commons) to function properly. I didn't find the
migration instructions very helpful; I run FastilyBot on a Raspberry Pi, and needless to say it would be grossly impractical for me to perform a "join" in the bot's code.
Is there going to be a replacement for this functionality?
Fastily
[4] was made to figure out common use cases and possibilities to enable them again.
...
I just want to highlight this ^ thing Joaquin said and mention that our team (Data Engineering) is also participating in brainstorming ways to bring back not just cross-wiki joins but better datasets to run these queries. We have some good ideas, so please do participate in the task and give us more input so we can pick the best solution quickly.
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly labs-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation