jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/432772 )
Change subject: [IMPR] use roundrobin_generators to combine generators when limit is given ......................................................................
[IMPR] use roundrobin_generators to combine generators when limit is given
Change-Id: I11957c120c911b701581e60d6c3b965980478999 --- M pywikibot/pagegenerators/__init__.py M pywikibot/pagegenerators/_factory.py 2 files changed, 24 insertions(+), 4 deletions(-)
Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified
diff --git a/pywikibot/pagegenerators/__init__.py b/pywikibot/pagegenerators/__init__.py index fa30b7d..4608b6c 100644 --- a/pywikibot/pagegenerators/__init__.py +++ b/pywikibot/pagegenerators/__init__.py @@ -432,8 +432,10 @@
-intersect Work on the intersection of all the provided generators.
--limit When used with any other argument -limit:n specifies a set - of pages, work on no more than n pages in total. +-limit When used with any other argument ``-limit:n`` + specifies a set of pages, work on no more than n + pages in total. If used with multiple generators, + pages are yielded in a roundrobin way.
-namespaces Filter the page generator to only yield pages in the -namespace specified namespaces. Separate multiple namespace diff --git a/pywikibot/pagegenerators/_factory.py b/pywikibot/pagegenerators/_factory.py index 0b42c73..f32c8e3 100644 --- a/pywikibot/pagegenerators/_factory.py +++ b/pywikibot/pagegenerators/_factory.py @@ -54,7 +54,11 @@ WikidataSPARQLPageGenerator, ) from pywikibot.tools.collections import DequeGenerator -from pywikibot.tools.itertools import filter_unique, intersect_generators +from pywikibot.tools.itertools import ( + filter_unique, + intersect_generators, + roundrobin_generators, +)
HANDLER_RETURN_TYPE = Union[None, bool, Iterable['pywikibot.page.BasePage']] @@ -200,6 +204,10 @@
.. versionchanged:: 7.3 set the instance variable :attr:`is_preloading` to True or False. + .. versionchanged:: 8.0 + if ``limit`` option is set and multiple generators are given, + pages are yieded in a :func:`roundrobin + <tools.itertools.roundrobin_generators>` way.
:param gen: Another generator to be combined with :param preload: preload pages using PreloadingGenerator @@ -247,7 +255,8 @@ # By definition no duplicates are possible. dupfiltergen = intersect_generators(*self.gens) else: - dupfiltergen = _filter_unique_pages(itertools.chain(*self.gens)) + combine = roundrobin_generators if self.limit else itertools.chain + dupfiltergen = _filter_unique_pages(combine(*self.gens))
# Add on subpage filter generator if self.subpage_max_depth is not None: