Without the screenshot this time....
On Mar 28, 2023, at 2:01 PM, Roy Smith roy@panix.com wrote:
Hmmm. What I'm doing requires Page.expand_text(), which looks like it does a Page.get() followed by a Site.expand_text(), and it's the later which actually takes most of the time. That becomes an action=expandtemplates API call https://www.mediawiki.org/w/api.php?action=help&modules=expandtemplates, which I don't see any way to batch.
<Screen Shot 2023-03-28 at 1.55.55 PM.png>
On Mar 28, 2023, at 1:04 PM, Kunal Mehta <legoktm@debian.org mailto:legoktm@debian.org> wrote:
Hi,
On 3/27/23 15:57, Roy Smith wrote:
I need to issue a bunch of Page.get() requests in parallel.
Please don't. From <https://www.mediawiki.org/wiki/API:Etiquette#Request_limit https://www.mediawiki.org/wiki/API:Etiquette#Request_limit>:
"Making your requests in series rather than in parallel, by waiting for one request to finish before sending a new request, should result in a safe request rate."
Instead of making parallel requests, you should make batched requests, which is how the preloading stuff Xqt mentioned works.
-- Kunal / Legoktm _______________________________________________ pywikibot mailing list -- pywikibot@lists.wikimedia.org mailto:pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/message/RXANREWJYNQ3AI2MDU4K4WR67D5W4KCH/ To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org mailto:pywikibot-leave@lists.wikimedia.org
Hi,
On 3/28/23 14:07, Roy Smith wrote:
Hmmm. What I'm doing requires Page.expand_text(), which looks like it does a Page.get() followed by a Site.expand_text(), and it's the later which actually takes most of the time. That becomes an action=expandtemplates API call https://www.mediawiki.org/w/api.php?action=help&modules=expandtemplates, which I don't see any way to batch.
Naively you can batch action=expandtemplates by putting unique markers in between each fragment of wikitext if you know the template will emit balanced wikitext (this part is key).
For example:
{{foo1}} --mycoolbot123-- {{foo2}} --mycoolbot123-- {{foo3}}
Then split the response on your marker and sync up with your list of pages :)
I suppose it's also worth asking what you're using expand_text() for in the first place, to see if there's a better way to do whatever it is you want to :)
For real concurrency I'll refer you to the blog post I wrote last year about this: https://blog.legoktm.com/2022/01/21/building-fast-wikipedia-bots-in-rust.html. Yes it's about Rust, but the same concepts can easily be replicated in Python.
-- Kunal / Legoktm
On Mar 28, 2023, at 9:09 PM, Kunal Mehta legoktm@debian.org wrote:
I suppose it's also worth asking what you're using expand_text() for in the first place, to see if there's a better way to do whatever it is you want to :)
That's a fair question.
What I'm doing is looking at DYK nominations to evaluate if they've been approved. Like so many wiki things, there's no formal definition, but the simple version is that I'm looking for "File:Symbol confirmed.svg". The problem is that it may not appear in the raw wikitext. An example is Bismarck Kuyon https://en.wikipedia.org/wiki/Template:Did_you_know_nominations/Bismarck_Kuyon. Looking at the page, it's easy to see the green checkmark indicating approval. But looking at the wikitext source, there's no such thing. What there is, is a {{DYK checklist}} template which invokes some Lua code that generates the checkmark based on the values in the other fields. The expand_text() forces that to get run on the server side.
From a machine-parsability point of view, it's insane. But I gotta work with what I've been given.
Ultimately, this is going to run as a bot. That fact that it takes a couple of minutes to evaluate all the nominations of interest isn't critical. I was doing an interactive web-based version for review purposes, and for that, waiting 2 minutes for the page to load sucked. But, I don't really need to do that, so I'll probably just go back to the serialized version and leave it at that.
One optimization I can see is that I only really need to do the expand_text() on the subset of nominations which use {{DYK checklist}}, and not even all of those (sometimes it's possible to determine the approval state entirely from the text following the {{DYK checklist}}). That will add a bit more complexity, which I was trying to avoid.
Even deeper down the complexity rathole, I could re-implement the Lua logic on the client side and avoid the expand_text() completely. I believe that's what some existing bots, such as WugBot do. But I really didn't want to go there.
I did a little reading about your mwbot-rs project. At one point, I was actually kind of excited about Rust and might have joined you just for the excuse to learn it. Maybe some day. I am totally about your goal of "sustainable development of bots and tools". We've got so many tools (some of which important processes like DYK are totally dependent on) which are, frankly, a mess of single-purpose code which can't be easily reused for anything else. What I've been trying to do with dyk-tools is create a toolkit of reusable components which other people can build upon. But I seem to be spending most of my time working around silly things like the {{DYK checklist}} stuff.
Anyway, I hope that answers your question :-)
BTW, I've mentioned this before, but I really can't recommend viztracer https://github.com/gaogaotiantian/viztracer highly enough as a performance analysis tool. At one level, it's just cProfile on steroids, but with a snazzy graphical front end. It's what let me figure out that it was expand(), not get(), which was the most expensive. I uploaded a screenshot to commons. https://commons.wikimedia.org/wiki/File:Screen_Shot_of_viztracer_output.png
If you are looking to check for image/file usage it’s better to query the api for just image used images instead of trying to parse wiki text
On Tue, Mar 28, 2023 at 10:29 PM Roy Smith roy@panix.com wrote:
On Mar 28, 2023, at 9:09 PM, Kunal Mehta legoktm@debian.org wrote:
I suppose it's also worth asking what you're using expand_text() for in the first place, to see if there's a better way to do whatever it is you want to :)
That's a fair question.
What I'm doing is looking at DYK nominations to evaluate if they've been approved. Like so many wiki things, there's no formal definition, but the simple version is that I'm looking for "File:Symbol confirmed.svg". The problem is that it may not appear in the raw wikitext. An example is Bismarck Kuyon https://en.wikipedia.org/wiki/Template:Did_you_know_nominations/Bismarck_Kuyon. Looking at the page, it's easy to see the green checkmark indicating approval. But looking at the wikitext source, there's no such thing. What there is, is a {{DYK checklist}} template which invokes some Lua code that generates the checkmark based on the values in the other fields. The expand_text() forces that to get run on the server side.
From a machine-parsability point of view, it's insane. But I gotta work with what I've been given.
Ultimately, this is going to run as a bot. That fact that it takes a couple of minutes to evaluate all the nominations of interest isn't critical. I was doing an interactive web-based version for review purposes, and for that, waiting 2 minutes for the page to load sucked. But, I don't really need to do that, so I'll probably just go back to the serialized version and leave it at that.
One optimization I can see is that I only really need to do the expand_text() on the subset of nominations which use {{DYK checklist}}, and not even all of those (sometimes it's possible to determine the approval state entirely from the text following the {{DYK checklist}}). That will add a bit more complexity, which I was trying to avoid.
Even deeper down the complexity rathole, I could re-implement the Lua logic on the client side and avoid the expand_text() completely. I believe that's what some existing bots, such as WugBot do. But I really didn't want to go there.
I did a little reading about your mwbot-rs project. At one point, I was actually kind of excited about Rust and might have joined you just for the excuse to learn it. Maybe some day. I am totally about your goal of "sustainable development of bots and tools". We've got so many tools (some of which important processes like DYK are totally dependent on) which are, frankly, a mess of single-purpose code which can't be easily reused for anything else. What I've been trying to do with dyk-tools is create a toolkit of reusable components which other people can build upon. But I seem to be spending most of my time working around silly things like the {{DYK checklist}} stuff.
Anyway, I hope that answers your question :-)
BTW, I've mentioned this before, but I really can't recommend viztracer https://github.com/gaogaotiantian/viztracer highly enough as a performance analysis tool. At one level, it's just cProfile on steroids, but with a snazzy graphical front end. It's what let me figure out that it was expand(), not get(), which was the most expensive. I uploaded a screenshot to commons. https://commons.wikimedia.org/wiki/File:Screen_Shot_of_viztracer_output.png
pywikibot mailing list -- pywikibot@lists.wikimedia.org Public archives at https://lists.wikimedia.org/hyperkitty/list/pywikibot@lists.wikimedia.org/me... To unsubscribe send an email to pywikibot-leave@lists.wikimedia.org
Been there, done that. I assume you're talking about Page.imagelinks(). That gets you the images in arbitrary order. I need to get them in the order they're in the wikitext. A given nomination can be approved, dis-appoved, re-approved, etc, as people squabble over the nomination. The last one is the one that counts.
On Mar 28, 2023, at 10:57 PM, John phoenixoverride@gmail.com wrote:
If you are looking to check for image/file usage it’s better to query the api for just image used images instead of trying to parse wiki text
On Tue, Mar 28, 2023 at 10:29 PM Roy Smith <roy@panix.com mailto:roy@panix.com> wrote: