jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/348233 )

Change subject: Do not exceed API limit of pageids in BaseSite.preloadpages ......................................................................

Do not exceed API limit of pageids in BaseSite.preloadpages

AllpagesGenerator (and possibly others) returns pages with their ids. When reused in PreloadingGenerator, this generates a warning and truncates the series of pages to the limit.

Bug: T78333 Bug: T161783 Change-Id: I240b0f998d5ef9c6438338da9572bdde265a3699 --- M pywikibot/site.py M scripts/category_redirect.py 2 files changed, 10 insertions(+), 4 deletions(-)

Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved

Objections: Mpaa: There's a problem with this change, please improve

diff --git a/pywikibot/site.py b/pywikibot/site.py index a16bdea..e5259c0 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -3257,7 +3257,14 @@ next_prio = 0 rvgen = api.PropertyGenerator(props, site=self) rvgen.set_maximum_items(-1) # suppress use of "rvlimit" parameter - if len(pageids) == len(sublist): + + parameter = self._paraminfo.parameter('query+info', 'prop') + if self.logged_in() and self.has_right('apihighlimits'): + max_ids = int(parameter['highlimit']) + else: + max_ids = int(parameter['limit']) # T78333, T161783 + + if len(pageids) == len(sublist) and len(set(pageids)) <= max_ids: # only use pageids if all pages have them rvgen.request['pageids'] = set(pageids) else: diff --git a/scripts/category_redirect.py b/scripts/category_redirect.py index 65aae8d..d616d4f 100755 --- a/scripts/category_redirect.py +++ b/scripts/category_redirect.py @@ -223,9 +223,8 @@ comment = i18n.twtranslate(self.site, self.redir_comment)

# generator yields all hard redirect pages in namespace 14 - for page in pagegenerators.PreloadingGenerator( - self.site.allpages(namespace=14, filterredir=True), - groupsize=250): + for page in self.site.allpages(namespace=14, filterredir=True, + content=True): if page.isCategoryRedirect(): # this is already a soft-redirect, so skip it (for now) continue