jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428417 )
Change subject: [fix] charset is a keyword argument of request
......................................................................
[fix] charset is a keyword argument of request
Change-Id: I3dfc91da47bcded58509d153f0b3074c6d251ec6
---
M pywikibot/comms/http.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Dalba: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py
index 0ac5187..4c096e0 100644
--- a/pywikibot/comms/http.py
+++ b/pywikibot/comms/http.py
@@ -285,7 +285,7 @@
@type site: L{pywikibot.site.BaseSite}
@param uri: the URI to retrieve
@type uri: str
- @param charset: Either a valid charset (usable for str.decode()) or None
+ @keyword charset: Either a valid charset (usable for str.decode()) or None
to automatically chose the charset from the returned header (defaults
to latin-1)
@type charset: CodecInfo, str, None
--
To view, visit https://gerrit.wikimedia.org/r/428417
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3dfc91da47bcded58509d153f0b3074c6d251ec6
Gerrit-Change-Number: 428417
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428423 )
Change subject: [cleanup] Throttle.lastwait was never used and can be deprecated
......................................................................
[cleanup] Throttle.lastwait was never used and can be deprecated
Change-Id: I3b923b880260a056ac5130d097c9ac24f3c87728
---
M pywikibot/throttle.py
1 file changed, 8 insertions(+), 2 deletions(-)
Approvals:
Dalba: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/throttle.py b/pywikibot/throttle.py
index 40022fb..ebad441 100644
--- a/pywikibot/throttle.py
+++ b/pywikibot/throttle.py
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
"""Mechanics to slow down wiki read and/or write rate."""
#
-# (C) Pywikibot team, 2008-2017
+# (C) Pywikibot team, 2008-2018
#
# Distributed under the terms of the MIT license.
#
@@ -13,6 +13,7 @@
import pywikibot
from pywikibot import config
+from pywikibot.tools import deprecated
_logger = "wiki.throttle"
@@ -64,7 +65,6 @@
# Free the process id after this many seconds:
self.releasepid = 1200
- self.lastwait = 0.0
self.delay = 0
self.checktime = 0
self.multiplydelay = multiplydelay
@@ -72,6 +72,12 @@
self.checkMultiplicity()
self.setDelays()
+ @deprecated
+ @property
+ def lastwait(self):
+ """DEPRECATED property."""
+ return 0.0
+
def checkMultiplicity(self):
"""Count running processes for site and set process_multiplicity."""
global pid
--
To view, visit https://gerrit.wikimedia.org/r/428423
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I3b923b880260a056ac5130d097c9ac24f3c87728
Gerrit-Change-Number: 428423
Gerrit-PatchSet: 3
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/426819 )
Change subject: [PEP 479] Change StopIteration handling inside generators
......................................................................
[PEP 479] Change StopIteration handling inside generators
User.uploadedImages:
just return instead of raising StopIteration
interwiki.generateMore:
StopIteration was raised only to leave the the outer for loop.
Implement a more suitable way to escape from that loop:
Use for loop for the pageGenerator which stops looping if the generator
is empty. In that case the outer for loop could be finished. set
self.pageGenerator to None then.
The IOError is from old compat code which used their
TextfileGenerator to retrieve pages from the text file. Now pickle is
used and IOError is handled when loading from file.
https://www.python.org/dev/peps/pep-0479/#why-not-fix-all-next-methods
Change-Id: Ic342346698d01f8dd0050a2558e689edaebdfb6c
---
M pywikibot/page.py
M scripts/interwiki.py
2 files changed, 51 insertions(+), 53 deletions(-)
Approvals:
Dalba: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/page.py b/pywikibot/page.py
index 97c6681..b5b0396 100644
--- a/pywikibot/page.py
+++ b/pywikibot/page.py
@@ -3589,9 +3589,8 @@
@type total: int
"""
if not self.isRegistered():
- raise StopIteration
- for item in self.logevents(
- logtype='upload', total=total):
+ return
+ for item in self.logevents(logtype='upload', total=total):
yield (item.page(),
unicode(item.timestamp()),
item.comment(),
diff --git a/scripts/interwiki.py b/scripts/interwiki.py
index 8235e14..0333585 100755
--- a/scripts/interwiki.py
+++ b/scripts/interwiki.py
@@ -2063,60 +2063,59 @@
'NOTE: Number of pages queued is {0}, trying to add {1} more.'
.format(len(self.subjects), number))
for i in range(number):
- try:
- while True:
+ for page in self.pageGenerator:
+ if page in self.conf.skip:
+ pywikibot.output('Skipping: {0} is in the skip list'
+ .format(page))
+ continue
+ if self.conf.skipauto:
+ dictName, year = page.autoFormat()
+ if dictName is not None:
+ pywikibot.output(
+ 'Skipping: {0} is an auto entry {1}({2})'
+ .format(page, dictName, year))
+ continue
+ if self.conf.parenthesesonly:
+ # Only yield pages that have ( ) in titles
+ if '(' not in page.title():
+ continue
+ if page.isTalkPage():
+ pywikibot.output('Skipping: {} is a talk page'
+ .format(page))
+ continue
+ if page.namespace() == 10:
try:
- page = next(self.pageGenerator)
- except IOError:
- pywikibot.output(u'IOError occurred; skipping')
+ tmpl, loc = moved_links[page.site.code]
+ del tmpl
+ except KeyError:
+ loc = None
+ if loc is not None and loc in page.title():
+ pywikibot.output(
+ 'Skipping: {} is a templates subpage'
+ .format(page.title()))
continue
- if page in self.conf.skip:
- pywikibot.output('Skipping: {0} is in the skip list'
- .format(page))
- continue
- if self.conf.skipauto:
- dictName, year = page.autoFormat()
- if dictName is not None:
- pywikibot.output(
- 'Skipping: {0} is an auto entry {1}({2})'
- .format(page, dictName, year))
- continue
- if self.conf.parenthesesonly:
- # Only yield pages that have ( ) in titles
- if "(" not in page.title():
- continue
- if page.isTalkPage():
- pywikibot.output(u'Skipping: %s is a talk page' % page)
- continue
- if page.namespace() == 10:
- try:
- tmpl, loc = moved_links[page.site.code]
- del tmpl
- except KeyError:
- loc = None
- if loc is not None and loc in page.title():
- pywikibot.output(
- 'Skipping: %s is a templates subpage'
- % page.title())
- continue
+ break
+ else: # generator stopped
+ break
+
+ if self.generateUntil:
+ until = self.generateUntil
+ page_namespace = (
+ page.site.namespaces[int(page.namespace())])
+ if page_namespace.case == 'first-letter':
+ until = first_upper(until)
+ if page.title(withNamespace=False) > until:
break
- if self.generateUntil:
- until = self.generateUntil
- page_namespace = (
- page.site.namespaces[int(page.namespace())])
- if page_namespace.case == 'first-letter':
- until = first_upper(until)
- if page.title(withNamespace=False) > until:
- raise StopIteration
- self.add(page, hints=self.conf.hints)
- self.generated += 1
- if self.generateNumber:
- if self.generated >= self.generateNumber:
- raise StopIteration
- except StopIteration:
- self.pageGenerator = None
- break
+ self.add(page, hints=self.conf.hints)
+ self.generated += 1
+ if self.generateNumber:
+ if self.generated >= self.generateNumber:
+ break
+ else:
+ return
+ # for loop was exited by break statement
+ self.pageGenerator = None
def firstSubject(self):
"""Return the first subject that is still being worked on."""
--
To view, visit https://gerrit.wikimedia.org/r/426819
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic342346698d01f8dd0050a2558e689edaebdfb6c
Gerrit-Change-Number: 426819
Gerrit-PatchSet: 4
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428263 )
Change subject: page.Category.articles: Fix the equivalant of startFrom in core
......................................................................
page.Category.articles: Fix the equivalant of startFrom in core
The way `startFrom` works in compat is equal to `startprefix` in core.[1]
`startsort`, which is deprecated in MW 1.24, needs to be a binary string,[2]
otherwise the user will get the wrong results because of the wrong start.
This means that compat's `startFrom` has no equivalant in core if the
MediaWiki version is less than 1.18.
[1]:
https://phabricator.wikimedia.org/diffusion/PWBO/browse/master/catlib.py;df…
[2]: https://www.mediawiki.org/wiki/API:Categorymembers
Change-Id: I10620e82d2d0792579889199ffa5e94a0cce0d07
---
M pywikibot/page.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Zhuyifei1999: Looks good to me, but someone else must approve
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/page.py b/pywikibot/page.py
index 97c6681..93d5f40 100644
--- a/pywikibot/page.py
+++ b/pywikibot/page.py
@@ -2798,7 +2798,7 @@
if total == 0:
return
- @deprecated_args(startFrom='startsort', step=None)
+ @deprecated_args(startFrom='startprefix', step=None)
def articles(self, recurse=False, total=None,
content=False, namespaces=None, sortby=None,
reverse=False, starttime=None, endtime=None,
--
To view, visit https://gerrit.wikimedia.org/r/428263
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I10620e82d2d0792579889199ffa5e94a0cce0d07
Gerrit-Change-Number: 428263
Gerrit-PatchSet: 1
Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428148 )
Change subject: CategorizedPageGenerator: Use startprefix parameter of category.articles
......................................................................
CategorizedPageGenerator: Use startprefix parameter of category.articles
The previous method of passing `start` to `startsort` was wrong. It does not
work as expected because `startsort` needs to be a binary string.[1]
For older versions of MediaWiki implement a workaround by iterating over
all category.articles. This is inefficient, but no better way found.
Remove `kwargs['sortby'] = 'sortkey'`, 'sortkey' is API's default.
[1]: https://www.mediawiki.org/wiki/API:Categorymembers
Bug: T74101
Bug: T143120
Change-Id: I7eece66e258659f84479fbdd1094a8df9ee69479
---
M pywikibot/pagegenerators.py
1 file changed, 12 insertions(+), 2 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py
index e503671..9071e1a 100644
--- a/pywikibot/pagegenerators.py
+++ b/pywikibot/pagegenerators.py
@@ -56,6 +56,7 @@
)
from pywikibot.logentries import LogEntryFactory
from pywikibot.proofreadpage import ProofreadPage
+from pywikibot.tools import MediaWikiVersion
if sys.version_info[0] > 2:
basestring = (str, )
@@ -1386,8 +1387,17 @@
'content': content, 'namespaces': namespaces,
}
if start:
- kwargs['sortby'] = 'sortkey'
- kwargs['startsort'] = start
+ if category.site.version() < MediaWikiVersion('1.18'):
+ kwargs.pop('total', None)
+ count = 0
+ for article in category.articles(**kwargs):
+ if article.title() >= start:
+ count += 1
+ yield article
+ if count == total:
+ return
+ return
+ kwargs['startprefix'] = start
for a in category.articles(**kwargs):
yield a
--
To view, visit https://gerrit.wikimedia.org/r/428148
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7eece66e258659f84479fbdd1094a8df9ee69479
Gerrit-Change-Number: 428148
Gerrit-PatchSet: 6
Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428069 )
Change subject: [IMPR] reduce code complexity of QueryGenerator.__iter__
......................................................................
[IMPR] reduce code complexity of QueryGenerator.__iter__
- Reduce cyclomatic complexity of submit from E37 to D23
and improve readability by splitting code in parts.
Change-Id: Iaa1427ca3a0afc6bcc16e8ff379dabf5853fd36c
---
M pywikibot/data/api.py
1 file changed, 73 insertions(+), 65 deletions(-)
Approvals:
Dalba: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py
index 1fed1f9..b9c54e0 100644
--- a/pywikibot/data/api.py
+++ b/pywikibot/data/api.py
@@ -2732,6 +2732,74 @@
value = str(value)
self.request[key] = value
+ def _handle_query_limit(self, prev_limit, new_limit, count, had_data):
+ """Handle query limit."""
+ if self.query_limit is None:
+ return prev_limit, new_limit
+
+ prev_limit = new_limit
+ if self.limit is None:
+ new_limit = self.query_limit
+ elif self.limit > 0:
+ if had_data:
+ # self.resultkey in data in last request.submit()
+ new_limit = min(self.query_limit, self.limit - count)
+ else:
+ # only "(query-)continue" returned. See Bug T74209.
+ # increase new_limit to advance faster until new
+ # useful data are found again.
+ new_limit = min(new_limit * 2, self.query_limit)
+ else:
+ new_limit = None
+
+ if new_limit and 'rvprop' in self.request \
+ and 'content' in self.request['rvprop']:
+ # queries that retrieve page content have lower limits
+ # Note: although API allows up to 500 pages for content
+ # queries, these sometimes result in server-side errors
+ # so use 250 as a safer limit
+ new_limit = min(new_limit, self.api_limit // 10, 250)
+
+ if new_limit is not None:
+ self.request[self.prefix + 'limit'] = str(new_limit)
+
+ if prev_limit != new_limit:
+ pywikibot.debug(
+ '{name}: query_limit: {query}, api_limit: {api}, '
+ 'limit: {limit}, new_limit: {new}, count: {count}\n'
+ '{name}: {prefix}limit: {value}'
+ .format(name=self.__class__.__name__,
+ query=self.query_limit,
+ api=self.api_limit,
+ limit=self.limit,
+ new=new_limit,
+ count=count,
+ prefix=self.prefix,
+ value=self.request[self.prefix + 'limit']),
+ _logger)
+ return prev_limit, new_limit
+
+ def _get_resultdata(self):
+ """Get resultdata and verify result."""
+ resultdata = keys = self.data['query'][self.resultkey]
+ if isinstance(resultdata, dict):
+ keys = list(resultdata.keys())
+ if 'results' in resultdata:
+ resultdata = resultdata['results']
+ elif 'pageids' in self.data['query']:
+ # this ensures that page data will be iterated
+ # in the same order as received from server
+ resultdata = [resultdata[k]
+ for k in self.data['query']['pageids']]
+ else:
+ resultdata = [resultdata[k]
+ for k in sorted(resultdata.keys())]
+ pywikibot.debug('{name} received {keys}; limit={limit}'
+ .format(name=self.__class__.__name__,
+ keys=keys, limit=self.limit),
+ _logger)
+ return resultdata
+
def __iter__(self):
"""Submit request and iterate the response based on self.resultkey.
@@ -2743,78 +2811,18 @@
count = 0
while True:
- if self.query_limit is not None:
- prev_limit = new_limit
- if self.limit is None:
- new_limit = self.query_limit
- elif self.limit > 0:
- if previous_result_had_data:
- # self.resultkey in data in last request.submit()
- new_limit = min(self.query_limit, self.limit - count)
- else:
- # only "(query-)continue" returned. See Bug T74209.
- # increase new_limit to advance faster until new
- # useful data are found again.
- new_limit = min(new_limit * 2, self.query_limit)
- else:
- new_limit = None
-
- if new_limit and \
- "rvprop" in self.request \
- and "content" in self.request["rvprop"]:
- # queries that retrieve page content have lower limits
- # Note: although API allows up to 500 pages for content
- # queries, these sometimes result in server-side errors
- # so use 250 as a safer limit
- new_limit = min(new_limit, self.api_limit // 10, 250)
- if new_limit is not None:
- self.request[self.prefix + "limit"] = str(new_limit)
- if prev_limit != new_limit:
- pywikibot.debug(
- u"%s: query_limit: %s, api_limit: %s, "
- u"limit: %s, new_limit: %s, count: %s"
- % (self.__class__.__name__,
- self.query_limit, self.api_limit,
- self.limit, new_limit, count),
- _logger)
- pywikibot.debug(
- u"%s: %s: %s"
- % (self.__class__.__name__,
- self.prefix + "limit",
- self.request[self.prefix + "limit"]),
- _logger)
+ prev_limit, new_limit = self._handle_query_limit(
+ prev_limit, new_limit, count, previous_result_had_data)
if not hasattr(self, "data"):
self.data = self.request.submit()
if not self.data or not isinstance(self.data, dict):
pywikibot.debug(
- u"%s: stopped iteration because no dict retrieved from api."
- % self.__class__.__name__,
+ '{}: stopped iteration because no dict retrieved from api.'
+ .format(self.__class__.__name__),
_logger)
return
if 'query' in self.data and self.resultkey in self.data["query"]:
- resultdata = self.data["query"][self.resultkey]
- if isinstance(resultdata, dict):
- pywikibot.debug(u"%s received %s; limit=%s"
- % (self.__class__.__name__,
- list(resultdata.keys()),
- self.limit),
- _logger)
- if "results" in resultdata:
- resultdata = resultdata["results"]
- elif "pageids" in self.data["query"]:
- # this ensures that page data will be iterated
- # in the same order as received from server
- resultdata = [resultdata[k]
- for k in self.data["query"]["pageids"]]
- else:
- resultdata = [resultdata[k]
- for k in sorted(resultdata.keys())]
- else:
- pywikibot.debug(u"%s received %s; limit=%s"
- % (self.__class__.__name__,
- resultdata,
- self.limit),
- _logger)
+ resultdata = self._get_resultdata()
if "normalized" in self.data["query"]:
self.normalized = {
item['to']: item['from']
--
To view, visit https://gerrit.wikimedia.org/r/428069
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaa1427ca3a0afc6bcc16e8ff379dabf5853fd36c
Gerrit-Change-Number: 428069
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428046 )
Change subject: [bugfix] Fix category title handling in makecat.py
......................................................................
[bugfix] Fix category title handling in makecat.py
The makecat.py was broken due to double colon in processed category
title. The script was not working at all and also the docs were missing
to the script (T187009).
Bug: T192706
Change-Id: I4d84e2d021c2ae85c11f45a898443f8c7b618312
---
M scripts/makecat.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/makecat.py b/scripts/makecat.py
index 0f647ad..4305fdd 100755
--- a/scripts/makecat.py
+++ b/scripts/makecat.py
@@ -217,7 +217,7 @@
summary = i18n.twtranslate(mysite, 'makecat-create',
{'cat': workingcatname})
workingcat = pywikibot.Category(mysite,
- u'%s:%s'
+ '%s%s'
% (mysite.namespaces.CATEGORY,
workingcatname))
filename = pywikibot.config.datafilepath(
--
To view, visit https://gerrit.wikimedia.org/r/428046
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4d84e2d021c2ae85c11f45a898443f8c7b618312
Gerrit-Change-Number: 428046
Gerrit-PatchSet: 2
Gerrit-Owner: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>