jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/588152 )
Change subject: [IMPR] No longer duplicate absolute_path in sys.path
......................................................................
[IMPR] No longer duplicate absolute_path in sys.path
No longer duplicate absolute_path in sys.path and overwrite
the first occurence later in run_python_file but just insert
and pop os.path.dirname(filename) there.
Change-Id: I84c58cc393165b12bf67bc0696826ae4baa8b258
---
M pwb.py
1 file changed, 3 insertions(+), 12 deletions(-)
Approvals:
Matěj Suchánek: Looks good to me, but someone else must approve
Huji: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pwb.py b/pwb.py
index e50d678..22fa3a8 100755
--- a/pwb.py
+++ b/pwb.py
@@ -88,11 +88,10 @@
# Set sys.argv and the first path element properly.
old_argv = sys.argv
old_argvu = pwb.argvu
- old_path0 = sys.path[0]
sys.argv = argv
pwb.argvu = argvu
- sys.path[0] = os.path.dirname(filename)
+ sys.path.insert(0, os.path.dirname(filename))
try:
with open(filename, 'rb') as f:
@@ -105,7 +104,7 @@
# Restore the old argv and path
sys.argv = old_argv
- sys.path[0] = old_path0
+ sys.path.pop(0)
pwb.argvu = old_argvu
# end of snippet from coverage
@@ -200,15 +199,6 @@
return not missing_requirements
-# Establish a normalised path for the directory containing pwb.py.
-# Either it is '.' if the user's current working directory is the same,
-# or it is the absolute path for the directory of pwb.py
-absolute_path = abspath(os.path.dirname(sys.argv[0]))
-
-# add absolute path because sys.path[0] is overridden
-# with os.path.dirname(filename) within run_python_file
-sys.path.insert(1, absolute_path)
-
if not check_modules():
sys.exit()
@@ -362,6 +352,7 @@
# a much more detailed implementation is in coverage's find_module.
# https://bitbucket.org/ned/coveragepy/src/default/coverage/execfile.py
cwd = abspath(os.getcwd())
+ absolute_path = abspath(os.path.dirname(sys.argv[0]))
if absolute_path == cwd:
absolute_filename = abspath(filename)[:len(cwd)]
if absolute_filename == cwd:
--
To view, visit https://gerrit.wikimedia.org/r/588152
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I84c58cc393165b12bf67bc0696826ae4baa8b258
Gerrit-Change-Number: 588152
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Huji <huji.huji(a)gmail.com>
Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot (75)
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/598269 )
Change subject: [IMPR] Throttle requests after ratelimits exceeded
......................................................................
[IMPR] Throttle requests after ratelimits exceeded
- add 'ratelimits' to uiprop to get the rate limits
- Handle 'ratelimited' response code:
* check for an appropriate rate limit for the current action
* calculate the delay for the next try
* wait that time or use the default wait cycle
- modify the Request.wait method to respect a delay as parameter
Bug: T253180
Change-Id: I28ca14e7b9d8c8296b978ac506955e15f8728090
---
M pywikibot/data/api.py
M pywikibot/site/__init__.py
2 files changed, 27 insertions(+), 5 deletions(-)
Approvals:
JJMC89: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py
index cca8864..243dd7f 100644
--- a/pywikibot/data/api.py
+++ b/pywikibot/data/api.py
@@ -1912,6 +1912,23 @@
self.wait()
return True
+ def _ratelimited(self):
+ """Handle ratelimited warning."""
+ ratelimits = self.site.userinfo['ratelimits']
+ delay = None
+
+ ratelimit = ratelimits.get(self.action, {})
+ # find the lowest wait time for the given action
+ for limit in ratelimit.values():
+ seconds = limit['seconds']
+ hits = limit['hits']
+ delay = min(delay or seconds, seconds / hits)
+
+ if not delay:
+ pywikibot.warning(
+ 'No rate limit found for action {}'.format(self.action))
+ self.wait(delay)
+
def _bad_token(self, code):
"""Check for bad token."""
if (code != 'badtoken' or self.site._loginstatus
@@ -2047,6 +2064,10 @@
self.wait()
continue
+ if code == 'ratelimited':
+ self._ratelimited()
+ continue
+
# If readapidenied is returned try to login
if code == 'readapidenied' and self.site._loginstatus in (-3, -1):
self.site.login()
@@ -2100,14 +2121,15 @@
raise MaxlagTimeoutError(msg)
- def wait(self):
+ def wait(self, delay=None):
"""Determine how long to wait after a failed request."""
self.max_retries -= 1
if self.max_retries < 0:
raise TimeoutError('Maximum retries attempted without success.')
- pywikibot.warning('Waiting %s seconds before retrying.'
- % self.retry_wait)
- pywikibot.sleep(self.retry_wait)
+ delay = delay or self.retry_wait
+ pywikibot.warning('Waiting {:.1g} seconds before retrying.'
+ .format(delay))
+ pywikibot.sleep(delay)
# double the next wait, but do not exceed config.retry_max seconds
self.retry_wait = min(config.retry_max, self.retry_wait * 2)
diff --git a/pywikibot/site/__init__.py b/pywikibot/site/__init__.py
index 7dcff1c..7e3b5b7 100644
--- a/pywikibot/site/__init__.py
+++ b/pywikibot/site/__init__.py
@@ -2159,7 +2159,7 @@
uirequest = self._simple_request(
action='query',
meta='userinfo',
- uiprop='blockinfo|hasmsg|groups|rights'
+ uiprop='blockinfo|hasmsg|groups|rights|ratelimits'
)
uidata = uirequest.submit()
assert 'query' in uidata, \
--
To view, visit https://gerrit.wikimedia.org/r/598269
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I28ca14e7b9d8c8296b978ac506955e15f8728090
Gerrit-Change-Number: 598269
Gerrit-PatchSet: 4
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: JJMC89 <JJMC89.Wikimedia(a)gmail.com>
Gerrit-Reviewer: Multichill <maarten(a)mdammers.nl>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot (75)
Xqt has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/592604 )
Change subject: Revert "SiteDetectionTestCase.assertSite: Skip tests on ConnectionError"
......................................................................
Revert "SiteDetectionTestCase.assertSite: Skip tests on ConnectionError"
Do not hide exceptions during tests except they are already known for a
single test and cannot be solved shortly.
This reverts commit 12daad21d4c5fa06f0981d5a591a49fb998b9641.
Change-Id: I5c1384d1bb39834fbdb9fb1dbbb798053253f1c9
---
M tests/site_detect_tests.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
jenkins-bot: Verified
Xqt: Looks good to me, approved
diff --git a/tests/site_detect_tests.py b/tests/site_detect_tests.py
index f7b237a..bdc2039 100644
--- a/tests/site_detect_tests.py
+++ b/tests/site_detect_tests.py
@@ -40,7 +40,7 @@
"""
try:
self.assertIsInstance(MWSite(url), MWSite)
- except (ServerError, Timeout, ConnectionError) as e:
+ except (ServerError, Timeout) as e:
self.skipTest(e)
def assertNoSite(self, url):
--
To view, visit https://gerrit.wikimedia.org/r/592604
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5c1384d1bb39834fbdb9fb1dbbb798053253f1c9
Gerrit-Change-Number: 592604
Gerrit-PatchSet: 3
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot (75)
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/594175 )
Change subject: [doc] Update pickle protocol doc in config2.py
......................................................................
[doc] Update pickle protocol doc in config2.py
Change-Id: If1dcd91cbf704895b9e04a6bbc821d8affab1a9b
---
M pywikibot/config2.py
1 file changed, 4 insertions(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/config2.py b/pywikibot/config2.py
index b5e5a1c..79860ac 100644
--- a/pywikibot/config2.py
+++ b/pywikibot/config2.py
@@ -879,9 +879,12 @@
# Pickle protocol version to use for storing dumps.
# This config variable is not used for loading dumps.
+# Version 0 is a more or less human-readable protocol
# Version 2 is common to both Python 2 and 3, and should
# be used when dumps are accessed by both versions.
-# Version 4 is only available for Python 3.4
+# Version 3 is only available for Python 3
+# Version 4 is only available for Python 3.4+
+# Version 5 was added with Python 3.8
pickle_protocol = 2
# ============================
--
To view, visit https://gerrit.wikimedia.org/r/594175
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If1dcd91cbf704895b9e04a6bbc821d8affab1a9b
Gerrit-Change-Number: 594175
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot (75)
Gerrit-CC: Dvorapa <dvorapa(a)seznam.cz>
Xqt has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/598180 )
Change subject: [cleanup] warnfile.py wasn't ported to core
......................................................................
[cleanup] warnfile.py wasn't ported to core
warnfile.py wasn't ported to core and the related Task T59097 was
declined. Remove the remaining codes from interwiki.py because it
is not functional.
Change-Id: I822fb44361d3691bc6326642ac5ca8dca3f979b5
---
M scripts/interwiki.py
1 file changed, 2 insertions(+), 32 deletions(-)
Approvals:
Huji: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/interwiki.py b/scripts/interwiki.py
index 9fd6a7d..f76aa75 100755
--- a/scripts/interwiki.py
+++ b/scripts/interwiki.py
@@ -51,14 +51,6 @@
pages, continue alphabetically starting at the last of the
dumped pages. The dump file will be subsequently removed.
- -warnfile: used as -warnfile:filename, reads all warnings from the
- given file that apply to the home wiki language,
- and read the rest of the warning as a hint. Then
- treats all the mentioned pages. A quicker way to
- implement warnfile suggestions without verifying them
- against the live wiki is using the warnfile.py
- script.
-
¶ms;
Additionally, these arguments can be used to restrict the bot to certain pages:
@@ -2243,23 +2235,6 @@
return True
-def readWarnfile(filename, bot):
- """Read old interlanguage conflicts."""
- import warnfile
- reader = warnfile.WarnfileReader(filename)
- # we won't use removeHints
- (hints, removeHints) = reader.getHints()
- for page, pagelist in hints.items():
- # The WarnfileReader gives us a list of pagelinks, but
- # titletranslate.py expects a list of strings, so we convert it back.
- # TODO: This is a quite ugly hack, in the future we should maybe make
- # titletranslate expect a list of pagelinks.
- hintStrings = ['{}:{}'.format(hintedPage.site.lang,
- hintedPage.title())
- for hintedPage in pagelist]
- bot.add(page, hints=hintStrings)
-
-
def page_empty_check(page):
"""
Return True if page should be skipped as it is almost empty.
@@ -2301,7 +2276,6 @@
namespaces = []
number = None
until = None
- warnfile = None
# a normal PageGenerator (which doesn't give hints, only Pages)
hintlessPageGen = None
optContinue = False
@@ -2319,9 +2293,8 @@
for arg in local_args:
if iwconf.readOptions(arg):
continue
- elif arg.startswith('-warnfile:'):
- warnfile = arg[10:]
- elif arg.startswith('-years'):
+
+ if arg.startswith('-years'):
# Look if user gave a specific year at which to start
# Must be a natural number or negative integer.
if len(arg) > 7 and (arg[7:].isdigit()
@@ -2436,9 +2409,6 @@
hintlessPageGen, namespaces, site)
# we'll use iter() to create make a next() function available.
bot.setPageGenerator(iter(hintlessPageGen), number=number, until=until)
- elif warnfile:
- # TODO: filter namespaces if -namespace parameter was used
- readWarnfile(warnfile, bot)
else:
if not singlePageTitle and not opthintsonly:
singlePageTitle = pywikibot.input('Which page to check:')
--
To view, visit https://gerrit.wikimedia.org/r/598180
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I822fb44361d3691bc6326642ac5ca8dca3f979b5
Gerrit-Change-Number: 598180
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: ChongDae <cdpark(a)gmail.com>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Huji <huji.huji(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot (75)
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/598199 )
Change subject: [IMPR] Enable tuple for regex parameter in RegexFilter
......................................................................
[IMPR] Enable tuple for regex parameter in RegexFilter
Change-Id: I52c4554c383a43122656687b37d487e0de74062e
---
M pywikibot/pagegenerators.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Huji: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py
index 1521617..9675205 100644
--- a/pywikibot/pagegenerators.py
+++ b/pywikibot/pagegenerators.py
@@ -1895,7 +1895,7 @@
def __precompile(cls, regex, flag):
"""Precompile the regex list if needed."""
# Enable multiple regexes
- if not isinstance(regex, list):
+ if not isinstance(regex, (list, tuple)):
regex = [regex]
# Test if regex is already compiled.
# We assume that all list components have the same type
--
To view, visit https://gerrit.wikimedia.org/r/598199
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I52c4554c383a43122656687b37d487e0de74062e
Gerrit-Change-Number: 598199
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Huji <huji.huji(a)gmail.com>
Gerrit-Reviewer: jenkins-bot (75)
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/598157 )
Change subject: [IMPR] Check for missing generator after setup() call
......................................................................
[IMPR] Check for missing generator after setup() call
Bot.setup may create the generator.
Therefore check for it after setup()
call in front of the loop.
Change-Id: I73e01d3526b6e3248a23333d4f72866f7f187232
---
M pywikibot/bot.py
1 file changed, 2 insertions(+), 2 deletions(-)
Approvals:
Huji: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/bot.py b/pywikibot/bot.py
index 4d1cfcc..e234c09 100644
--- a/pywikibot/bot.py
+++ b/pywikibot/bot.py
@@ -1385,16 +1385,16 @@
@raise AssertionError: "page" is not a pywikibot.page.BasePage object
"""
self._start_ts = pywikibot.Timestamp.now()
+ self.setup()
+
if not hasattr(self, 'generator'):
raise NotImplementedError('Variable %s.generator not set.'
% self.__class__.__name__)
-
if PY2:
# Python 2 does not clear previous exceptions and method `exit`
# relies on sys.exc_info returning exceptions occurring in `run`.
sys.exc_clear()
- self.setup()
try:
for item in self.generator:
# preprocessing of the page
--
To view, visit https://gerrit.wikimedia.org/r/598157
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I73e01d3526b6e3248a23333d4f72866f7f187232
Gerrit-Change-Number: 598157
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Huji <huji.huji(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot (75)