jenkins-bot submitted this change.

View Change


Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified
[IMPR] use roundrobin_generators to combine generators when limit is given

Change-Id: I11957c120c911b701581e60d6c3b965980478999
---
M pywikibot/pagegenerators/__init__.py
M pywikibot/pagegenerators/_factory.py
2 files changed, 24 insertions(+), 4 deletions(-)

diff --git a/pywikibot/pagegenerators/__init__.py b/pywikibot/pagegenerators/__init__.py
index fa30b7d..4608b6c 100644
--- a/pywikibot/pagegenerators/__init__.py
+++ b/pywikibot/pagegenerators/__init__.py
@@ -432,8 +432,10 @@

-intersect Work on the intersection of all the provided generators.

--limit When used with any other argument -limit:n specifies a set
- of pages, work on no more than n pages in total.
+-limit When used with any other argument ``-limit:n``
+ specifies a set of pages, work on no more than n
+ pages in total. If used with multiple generators,
+ pages are yielded in a roundrobin way.

-namespaces Filter the page generator to only yield pages in the
-namespace specified namespaces. Separate multiple namespace
diff --git a/pywikibot/pagegenerators/_factory.py b/pywikibot/pagegenerators/_factory.py
index 0b42c73..f32c8e3 100644
--- a/pywikibot/pagegenerators/_factory.py
+++ b/pywikibot/pagegenerators/_factory.py
@@ -54,7 +54,11 @@
WikidataSPARQLPageGenerator,
)
from pywikibot.tools.collections import DequeGenerator
-from pywikibot.tools.itertools import filter_unique, intersect_generators
+from pywikibot.tools.itertools import (
+ filter_unique,
+ intersect_generators,
+ roundrobin_generators,
+)


HANDLER_RETURN_TYPE = Union[None, bool, Iterable['pywikibot.page.BasePage']]
@@ -200,6 +204,10 @@

.. versionchanged:: 7.3
set the instance variable :attr:`is_preloading` to True or False.
+ .. versionchanged:: 8.0
+ if ``limit`` option is set and multiple generators are given,
+ pages are yieded in a :func:`roundrobin
+ <tools.itertools.roundrobin_generators>` way.

:param gen: Another generator to be combined with
:param preload: preload pages using PreloadingGenerator
@@ -247,7 +255,8 @@
# By definition no duplicates are possible.
dupfiltergen = intersect_generators(*self.gens)
else:
- dupfiltergen = _filter_unique_pages(itertools.chain(*self.gens))
+ combine = roundrobin_generators if self.limit else itertools.chain
+ dupfiltergen = _filter_unique_pages(combine(*self.gens))

# Add on subpage filter generator
if self.subpage_max_depth is not None:

To view, visit change 432772. To unsubscribe, or for help writing mail filters, visit settings.

Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I11957c120c911b701581e60d6c3b965980478999
Gerrit-Change-Number: 432772
Gerrit-PatchSet: 3
Gerrit-Owner: Xqt <info@gno.de>
Gerrit-Reviewer: John Vandenberg <jayvdb@gmail.com>
Gerrit-Reviewer: Xqt <info@gno.de>
Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999@gmail.com>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged