I've got a list of O(100,000) pages. I want to get a feed of all the edits to those pages in real time. What's the most efficient way? The two strategies I can see are Site.recentchanges() and filter them on the client side, or put all the pages on my watchlist and call Site.watchlist_revs() periodically. Is there something more clever than those? Back-of-the-envelope, using numbers from https://stats.wikimedia.org/#/en.wikipedia.org, 6 million edits per month is 200k per day or 2-3 per second. So the recent_changes method doesn't seem too bad, but who knows...
The goal here is to build a real-time activity ticker to display at Wikipedia:Meetup/NYC/Wikipedia Day 2025 https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia_Day_2025. Maybe with some snazzy graphical front end. The set of pages is everything that's (recursively) a member of Category:New York City https://en.wikipedia.org/wiki/Category:New_York_City although I expect I'll probably apply some sort of filter like "has coordinates within the geographic limits of NYC". I'll prep the set off-line, so at the point where I'm running this, I'll have a pre-computed fixed set of pages.
A third approach: list them on 10 or 20 pages by 10000 or 5000 items, and monitor the connected changes.
Roy Smith roy@panix.com ezt írta (időpont: 2025. jan. 15., Sze, 1:26):
I've got a list of O(100,000) pages. I want to get a feed of all the edits to those pages in real time. What's the most efficient way? The two strategies I can see are Site.recentchanges() and filter them on the client side, or put all the pages on my watchlist and call Site.watchlist_revs() periodically. Is there something more clever than those? Back-of-the-envelope, using numbers from https://stats.wikimedia.org/#/en.wikipedia.org, 6 million edits per month is 200k per day or 2-3 per second. So the recent_changes method doesn't seem too bad, but who knows...
The goal here is to build a real-time activity ticker to display at Wikipedia:Meetup/NYC/Wikipedia Day 2025 https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia_Day_2025. Maybe with some snazzy graphical front end. The set of pages is everything that's (recursively) a member of Category:New York City https://en.wikipedia.org/wiki/Category:New_York_City although I expect I'll probably apply some sort of filter like "has coordinates within the geographic limits of NYC". I'll prep the set off-line, so at the point where I'm running this, I'll have a pre-computed fixed set of pages. _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
You can filter the recentchanges stream by several criteria so you reduce the workload, so this is the solution I would recommend.See https://www.mediawiki.org/wiki/API:RecentChanges for a full list; what seems relevant for your usecase: rcstart/rcend (for displaying only newer changes), rcnamespace (articles only), rctype (only edits), rctoponly (if you don't care about old changes).
Best, Strainu
În mie., 15 ian. 2025 la 02:26, Roy Smith roy@panix.com a scris:
I've got a list of O(100,000) pages. I want to get a feed of all the edits to those pages in real time. What's the most efficient way? The two strategies I can see are Site.recentchanges() and filter them on the client side, or put all the pages on my watchlist and call Site.watchlist_revs() periodically. Is there something more clever than those? Back-of-the-envelope, using numbers from https://stats.wikimedia.org/#/en.wikipedia.org, 6 million edits per month is 200k per day or 2-3 per second. So the recent_changes method doesn't seem too bad, but who knows...
The goal here is to build a real-time activity ticker to display at Wikipedia:Meetup/NYC/Wikipedia Day 2025 https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia_Day_2025. Maybe with some snazzy graphical front end. The set of pages is everything that's (recursively) a member of Category:New York City https://en.wikipedia.org/wiki/Category:New_York_City although I expect I'll probably apply some sort of filter like "has coordinates within the geographic limits of NYC". I'll prep the set off-line, so at the point where I'm running this, I'll have a pre-computed fixed set of pages. _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
You can also use EventStreams and ignore any edit that doesn't belong to your list: https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service
HTH
Am Mi., 15. Jan. 2025 um 09:24 Uhr schrieb Strainu strainu10@gmail.com:
You can filter the recentchanges stream by several criteria so you reduce the workload, so this is the solution I would recommend.See https://www.mediawiki.org/wiki/API:RecentChanges for a full list; what seems relevant for your usecase: rcstart/rcend (for displaying only newer changes), rcnamespace (articles only), rctype (only edits), rctoponly (if you don't care about old changes).
Best, Strainu
În mie., 15 ian. 2025 la 02:26, Roy Smith roy@panix.com a scris:
I've got a list of O(100,000) pages. I want to get a feed of all the edits to those pages in real time. What's the most efficient way? The two strategies I can see are Site.recentchanges() and filter them on the client side, or put all the pages on my watchlist and call Site.watchlist_revs() periodically. Is there something more clever than those? Back-of-the-envelope, using numbers from https://stats.wikimedia.org/#/en.wikipedia.org, 6 million edits per month is 200k per day or 2-3 per second. So the recent_changes method doesn't seem too bad, but who knows...
The goal here is to build a real-time activity ticker to display at Wikipedia:Meetup/NYC/Wikipedia Day 2025 https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia_Day_2025. Maybe with some snazzy graphical front end. The set of pages is everything that's (recursively) a member of Category:New York City https://en.wikipedia.org/wiki/Category:New_York_City although I expect I'll probably apply some sort of filter like "has coordinates within the geographic limits of NYC". I'll prep the set off-line, so at the point where I'm running this, I'll have a pre-computed fixed set of pages. _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
Cool, that looks like it might be what I want (although I'm wondering how expensive it will be if it doesn't support server-side filtering). I'm curious about the examples. At https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service, it gives an example:
change = next(iter(stream))
but the corresponding example at https://doc.wikimedia.org/pywikibot/stable/api_ref/pywikibot.comms.html#comm... just says:
change = next(stream)
Was there a change in the API at some point that required you to wrap it in iter()?
On Jan 15, 2025, at 8:06 AM, Amir Sarabadani ladsgroup@gmail.com wrote:
You can also use EventStreams and ignore any edit that doesn't belong to your list: https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service
HTH
Am Mi., 15. Jan. 2025 um 09:24 Uhr schrieb Strainu <strainu10@gmail.com mailto:strainu10@gmail.com>:
You can filter the recentchanges stream by several criteria so you reduce the workload, so this is the solution I would recommend.See https://www.mediawiki.org/wiki/API:RecentChanges for a full list; what seems relevant for your usecase: rcstart/rcend (for displaying only newer changes), rcnamespace (articles only), rctype (only edits), rctoponly (if you don't care about old changes).
Best, Strainu
În mie., 15 ian. 2025 la 02:26, Roy Smith <roy@panix.com mailto:roy@panix.com> a scris:
I've got a list of O(100,000) pages. I want to get a feed of all the edits to those pages in real time. What's the most efficient way? The two strategies I can see are Site.recentchanges() and filter them on the client side, or put all the pages on my watchlist and call Site.watchlist_revs() periodically. Is there something more clever than those? Back-of-the-envelope, using numbers from https://stats.wikimedia.org/#/en.wikipedia.org, 6 million edits per month is 200k per day or 2-3 per second. So the recent_changes method doesn't seem too bad, but who knows...
The goal here is to build a real-time activity ticker to display at Wikipedia:Meetup/NYC/Wikipedia Day 2025 https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia_Day_2025. Maybe with some snazzy graphical front end. The set of pages is everything that's (recursively) a member of Category:New York City https://en.wikipedia.org/wiki/Category:New_York_City although I expect I'll probably apply some sort of filter like "has coordinates within the geographic limits of NYC". I'll prep the set off-line, so at the point where I'm running this, I'll have a pre-computed fixed set of pages. _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org mailto:pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org mailto:pywikibot-leave@lists.wikimedia.org
pywikibot mailing list -- pywikibot@lists.wikimedia.org mailto:pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org mailto:pywikibot-leave@lists.wikimedia.org
-- Amir (he/him)
pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
I honestly don't know. I suggest trying on a small wiki and see how it goes
Am Mi., 15. Jan. 2025 um 16:09 Uhr schrieb Roy Smith roy@panix.com:
Cool, that looks like it might be what I want (although I'm wondering how expensive it will be if it doesn't support server-side filtering). I'm curious about the examples. At https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service, it gives an example:
change = next(iter(stream))
but the corresponding example at https://doc.wikimedia.org/pywikibot/stable/api_ref/pywikibot.comms.html#comm... just says:
change = next(stream)
Was there a change in the API at some point that required you to wrap it in iter()?
On Jan 15, 2025, at 8:06 AM, Amir Sarabadani ladsgroup@gmail.com wrote:
You can also use EventStreams and ignore any edit that doesn't belong to your list: https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service
HTH
Am Mi., 15. Jan. 2025 um 09:24 Uhr schrieb Strainu strainu10@gmail.com:
You can filter the recentchanges stream by several criteria so you reduce the workload, so this is the solution I would recommend.See https://www.mediawiki.org/wiki/API:RecentChanges for a full list; what seems relevant for your usecase: rcstart/rcend (for displaying only newer changes), rcnamespace (articles only), rctype (only edits), rctoponly (if you don't care about old changes).
Best, Strainu
În mie., 15 ian. 2025 la 02:26, Roy Smith roy@panix.com a scris:
I've got a list of O(100,000) pages. I want to get a feed of all the edits to those pages in real time. What's the most efficient way? The two strategies I can see are Site.recentchanges() and filter them on the client side, or put all the pages on my watchlist and call Site.watchlist_revs() periodically. Is there something more clever than those? Back-of-the-envelope, using numbers from https://stats.wikimedia.org/#/en.wikipedia.org, 6 million edits per month is 200k per day or 2-3 per second. So the recent_changes method doesn't seem too bad, but who knows...
The goal here is to build a real-time activity ticker to display at Wikipedia:Meetup/NYC/Wikipedia Day 2025 https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia_Day_2025. Maybe with some snazzy graphical front end. The set of pages is everything that's (recursively) a member of Category:New York City https://en.wikipedia.org/wiki/Category:New_York_City although I expect I'll probably apply some sort of filter like "has coordinates within the geographic limits of NYC". I'll prep the set off-line, so at the point where I'm running this, I'll have a pre-computed fixed set of pages. _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
-- Amir (he/him)
pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org