Hi,
On 3/28/23 14:07, Roy Smith wrote:
Hmmm. What I'm doing requires Page.expand_text(), which looks like it does a Page.get() followed by a Site.expand_text(), and it's the later which actually takes most of the time. That becomes an action=expandtemplates API call https://www.mediawiki.org/w/api.php?action=help&modules=expandtemplates, which I don't see any way to batch.
Naively you can batch action=expandtemplates by putting unique markers in between each fragment of wikitext if you know the template will emit balanced wikitext (this part is key).
For example:
{{foo1}} --mycoolbot123-- {{foo2}} --mycoolbot123-- {{foo3}}
Then split the response on your marker and sync up with your list of pages :)
I suppose it's also worth asking what you're using expand_text() for in the first place, to see if there's a better way to do whatever it is you want to :)
For real concurrency I'll refer you to the blog post I wrote last year about this: https://blog.legoktm.com/2022/01/21/building-fast-wikipedia-bots-in-rust.html. Yes it's about Rust, but the same concepts can easily be replicated in Python.
-- Kunal / Legoktm