Mostly for the sake of the archives, I ended up with this https://github.com/roysmith/dyk-tools/blob/60a0b1d5c6c5f7310a541f30388b898b4a906b10/dyk_tools/web/core.py#L69, which is pretty much straight out of the example in the python library manual:
def _is_approved(nom): return nom.is_approved() with ThreadPoolExecutor(max_workers=10) as executor: future_to_nom = { executor.submit(_is_approved, nom): nom for nom in nom_list.nominations() } for future in as_completed(future_to_nom): nom = future_to_nom[future] if future.result(): approved_noms.append(nom) else: unapproved_noms.append(nom)
One of the nagging questions in my mind as I was exploring this was whether APISite is thread-safe. I haven't found anything in the pywikibot docs which says one way or the other, but apparently it is.
I don't have a good feel for what max_workers should be. For what I'm doing, 10 seems to work well, taking about 2-3 seconds to process 70 nominations. The largest number of nominations I would ever expect to see is around 200. According to the library docs, max_workers defaults to 5 * number_of_processors. I don't actually have a clue what that works out to on a toolforge k8s instance, nor do I have any idea how the production enwiki would like it if I threw 100 parallel API requests at it all at once. So for now, I'll just hardwire it to 10.