jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428889 )
Change subject: Allow archivebot to archive even restricted pages
......................................................................
Allow archivebot to archive even restricted pages
The archive templates were recently added to templates which restricts
page to be edited, but it made archivebot not archive at all.
This patch excludes archive templates from restrictions and introduces a
method to filter out archive templates if archivebot is there.
TODO: This patch could be possibly expanded to some form of force
categories like:
page.save(force='always') x 'archive' x None
Change-Id: Iab249fd83f4f071b4694eba8d0b7d977ed22c3f8
---
M pywikibot/families/wikipedia_family.py
M pywikibot/family.py
M pywikibot/page.py
3 files changed, 25 insertions(+), 5 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/families/wikipedia_family.py b/pywikibot/families/wikipedia_family.py
index 0960980..b0234f6 100644
--- a/pywikibot/families/wikipedia_family.py
+++ b/pywikibot/families/wikipedia_family.py
@@ -205,11 +205,8 @@
self.edit_restricted_templates = {
'ar': ('تحرر',),
'bs': ('Izmjena u toku',),
- 'cs': ('Pracuje se', 'Archiv', 'Archiv Wikipedie',
- 'Archiv diskuse', 'Archivace start', 'Posloupnost archivů',
- 'Rfa-archiv-start', 'Rfc-archiv-start',),
- 'de': ('Inuse', 'In use', 'In bearbeitung', 'Inbearbeitung',
- 'Archiv',),
+ 'cs': ('Pracuje se',),
+ 'de': ('Inuse', 'In use', 'In bearbeitung', 'Inbearbeitung',),
'en': ('Inuse', 'In use'),
'fa': ('ویرایش',),
'fr': ('En cours', 'Plusieurs en cours', 'Correction en cours',
@@ -220,6 +217,15 @@
'zh': ('Inuse',),
}
+ # Archive templates that indicate an edit of non-archive bots
+ # should be avoided
+ self.archived_page_templates = {
+ 'cs': ('Archiv', 'Archiv Wikipedie', 'Archiv diskuse',
+ 'Archivace start', 'Posloupnost archivů',
+ 'Rfa-archiv-start', 'Rfc-archiv-start',),
+ 'de': ('Archiv',),
+ }
+
def get_known_families(self, site):
"""Override the family interwiki prefixes for each site."""
# In Swedish Wikipedia 's:' is part of page title not a family
diff --git a/pywikibot/family.py b/pywikibot/family.py
index ff18c20..936fbb9 100644
--- a/pywikibot/family.py
+++ b/pywikibot/family.py
@@ -730,6 +730,11 @@
# that indicate an edit should be avoided
self.edit_restricted_templates = {}
+ # A dict of tuples for different sites with names of archive
+ # templates that indicate an edit of non-archive bots
+ # should be avoided
+ self.archived_page_templates = {}
+
# A list of projects that share cross-project sessions.
if not hasattr(self, 'cross_projects'):
self.cross_projects = []
diff --git a/pywikibot/page.py b/pywikibot/page.py
index 91ee48c..6957e70 100644
--- a/pywikibot/page.py
+++ b/pywikibot/page.py
@@ -1171,6 +1171,15 @@
# multiple bots/nobots templates are allowed
restrictions = self.site.family.edit_restricted_templates.get(
self.site.code)
+ # also add archive templates for non-archive bots
+ if pywikibot.calledModuleName() != 'archivebot':
+ archived = self.site.family.archived_page_templates.get(
+ self.site.code)
+ if restrictions and archived:
+ restrictions += archived
+ elif archived:
+ restrictions = archived
+
for template, params in templates:
title = template.title(withNamespace=False)
if restrictions:
--
To view, visit https://gerrit.wikimedia.org/r/428889
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iab249fd83f4f071b4694eba8d0b7d977ed22c3f8
Gerrit-Change-Number: 428889
Gerrit-PatchSet: 5
Gerrit-Owner: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz>
Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/429193 )
Change subject: config2.py: Fix typo in makepath's docstring
......................................................................
config2.py: Fix typo in makepath's docstring
Change-Id: I7e137ae5a7c6bd5799ef198e4eb087bcc4def8c4
---
M pywikibot/config2.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/config2.py b/pywikibot/config2.py
index f68afb7..1ec7e93 100644
--- a/pywikibot/config2.py
+++ b/pywikibot/config2.py
@@ -904,7 +904,7 @@
def makepath(path, create=True):
"""Return a normalized absolute version of the path argument.
- If the given path already exists in the filesystem ore create is False
+ If the given path already exists in the filesystem or create is False
the filesystem is not modified.
Otherwise if create is True makepath creates directories along the given
path using the dirname() of the path. You may append a '/' to the path if
--
To view, visit https://gerrit.wikimedia.org/r/429193
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7e137ae5a7c6bd5799ef198e4eb087bcc4def8c4
Gerrit-Change-Number: 429193
Gerrit-PatchSet: 1
Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428157 )
Change subject: [IMPR] Enable makepath and datafilepath not to create the direcory
......................................................................
[IMPR] Enable makepath and datafilepath not to create the direcory
This could be used to check whether datafilepath(path) exists
without creating it.
Change-Id: I34943700bcd72541b921e3c63dd462e7fb5e2d3f
---
M pywikibot/config2.py
1 file changed, 22 insertions(+), 10 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/config2.py b/pywikibot/config2.py
index cf9e6b0..f68afb7 100644
--- a/pywikibot/config2.py
+++ b/pywikibot/config2.py
@@ -901,33 +901,45 @@
# #############################################
-def makepath(path):
+def makepath(path, create=True):
"""Return a normalized absolute version of the path argument.
- - if the given path already exists in the filesystem
- the filesystem is not modified.
- - otherwise makepath creates directories along the given path
- using the dirname() of the path. You may append
- a '/' to the path if you want it to be a directory path.
+ If the given path already exists in the filesystem ore create is False
+ the filesystem is not modified.
+ Otherwise if create is True makepath creates directories along the given
+ path using the dirname() of the path. You may append a '/' to the path if
+ you want it to be a directory path.
from holger(a)trillke.net 2002/03/18
+ @param path: path in the filesystem
+ @type path: str
+ @param create: create the directory if it is True. Otherwise do not change
+ the filesystem. Default is True.
+ @type create: bool
"""
dpath = os.path.normpath(os.path.dirname(path))
- if not os.path.exists(dpath):
+ if create and not os.path.exists(dpath):
os.makedirs(dpath)
return os.path.normpath(os.path.abspath(path))
-def datafilepath(*filename):
+def datafilepath(*filename, **kwargs):
"""Return an absolute path to a data file in a standard location.
Argument(s) are zero or more directory names, optionally followed by a
data file name. The return path is offset to config.base_dir. Any
- directories in the path that do not already exist are created.
+ directories in the path that do not already exist are created if create
+ is True, otherwise the filesystem keeps unchanged.
+ @param path: path in the filesystem
+ @type path: str
+ @keyword create: create the directory if it is True. Otherwise don't change
+ the filesystem. Default is True.
+ @type create: bool
"""
- return makepath(os.path.join(base_dir, *filename))
+ create = kwargs.get('create', True)
+ return makepath(os.path.join(base_dir, *filename), create=create)
def shortpath(path):
--
To view, visit https://gerrit.wikimedia.org/r/428157
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I34943700bcd72541b921e3c63dd462e7fb5e2d3f
Gerrit-Change-Number: 428157
Gerrit-PatchSet: 4
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/428408 )
Change subject: [cleanup] use textlib.NON_LATIN_DIGITS in date.py to prevent code duplication
......................................................................
[cleanup] use textlib.NON_LATIN_DIGITS in date.py to prevent code duplication
Change-Id: I9e330574a631772d2fc55feecf256fe5021d5096
---
M pywikibot/date.py
M pywikibot/textlib.py
2 files changed, 7 insertions(+), 5 deletions(-)
Approvals:
Framawiki: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/date.py b/pywikibot/date.py
index b965243..3e1d5f3 100644
--- a/pywikibot/date.py
+++ b/pywikibot/date.py
@@ -18,6 +18,7 @@
import re
import sys
+from pywikibot.textlib import NON_LATIN_DIGITS
from pywikibot.tools import first_lower, first_upper, deprecated
if sys.version_info[0] > 2:
@@ -260,27 +261,27 @@
# Helper for KN: digits representation
-_knDigits = u'೦೧೨೩೪೫೬೭೮೯'
+_knDigits = NON_LATIN_DIGITS['kn']
_knDigitsToLocal = {ord(unicode(i)): _knDigits[i] for i in range(10)}
_knLocalToDigits = {ord(_knDigits[i]): unicode(i) for i in range(10)}
# Helper for Urdu/Persian languages
-_faDigits = u'۰۱۲۳۴۵۶۷۸۹'
+_faDigits = NON_LATIN_DIGITS['fa']
_faDigitsToLocal = {ord(unicode(i)): _faDigits[i] for i in range(10)}
_faLocalToDigits = {ord(_faDigits[i]): unicode(i) for i in range(10)}
# Helper for HI:, MR:
-_hiDigits = u'०१२३४५६७८९'
+_hiDigits = NON_LATIN_DIGITS['hi']
_hiDigitsToLocal = {ord(unicode(i)): _hiDigits[i] for i in range(10)}
_hiLocalToDigits = {ord(_hiDigits[i]): unicode(i) for i in range(10)}
# Helper for BN:
-_bnDigits = u'০১২৩৪৫৬৭৮৯'
+_bnDigits = NON_LATIN_DIGITS['bn']
_bnDigitsToLocal = {ord(unicode(i)): _bnDigits[i] for i in range(10)}
_bnLocalToDigits = {ord(_bnDigits[i]): unicode(i) for i in range(10)}
# Helper for GU:
-_guDigits = u'૦૧૨૩૪૫૬૭૮૯'
+_guDigits = NON_LATIN_DIGITS['gu']
_guDigitsToLocal = {ord(unicode(i)): _guDigits[i] for i in range(10)}
_guLocalToDigits = {ord(_guDigits[i]): unicode(i) for i in range(10)}
diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py
index d637917..9e27312 100644
--- a/pywikibot/textlib.py
+++ b/pywikibot/textlib.py
@@ -145,6 +145,7 @@
NON_LATIN_DIGITS = {
'ckb': u'٠١٢٣٤٥٦٧٨٩',
'fa': u'۰۱۲۳۴۵۶۷۸۹',
+ 'hi': '०१२३४५६७८९',
'km': u'០១២៣៤៥៦៧៨៩',
'kn': u'೦೧೨೩೪೫೬೭೮೯',
'bn': u'০১২৩৪৫৬৭৮৯',
--
To view, visit https://gerrit.wikimedia.org/r/428408
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9e330574a631772d2fc55feecf256fe5021d5096
Gerrit-Change-Number: 428408
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/307127 )
Change subject: [IMPR] use retry-after value
......................................................................
[IMPR] use retry-after value
- 'retry-after' from response_header is a recommended minimum number of
seconds that the client should wait before retrying. As a first step
we use this value when retrying a request due to maxlag error.
- fallback to the lag value found by lagpattern in the api warning,
fallback to 5 seconds if both values are lost.
- Retrieve the 'retry-after' value via http.request.
- Throttle.lag should not shorten the lagtime anymore because we expect
the right value from upstream. But keep the max delay.
See https://www.mediawiki.org/wiki/Manual:Maxlag_parameter
Bug: T144023
Change-Id: Ie9666ba7654ab071bfeda2be53b29c3c568e508d
---
M pywikibot/comms/http.py
M pywikibot/data/api.py
M pywikibot/throttle.py
M tests/api_tests.py
4 files changed, 28 insertions(+), 17 deletions(-)
Approvals:
Framawiki: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py
index 0ac5187..995f7ff 100644
--- a/pywikibot/comms/http.py
+++ b/pywikibot/comms/http.py
@@ -305,8 +305,6 @@
r = fetch(uri, method, params, body, headers, **kwargs)
return r.text
- baseuri = site.base_url(uri)
-
kwargs.setdefault("disable_ssl_certificate_validation",
site.ignore_certificate_error())
@@ -314,11 +312,12 @@
headers = {}
format_string = None
else:
- format_string = headers.get('user-agent', None)
-
+ format_string = headers.get('user-agent')
headers['user-agent'] = user_agent(site, format_string)
+ baseuri = site.base_url(uri)
r = fetch(baseuri, method, params, body, headers, **kwargs)
+ site.throttle.retry_after = int(r.response_headers.get('retry-after', 0))
return r.text
diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py
index b9c54e0..274aba4 100644
--- a/pywikibot/data/api.py
+++ b/pywikibot/data/api.py
@@ -2077,11 +2077,12 @@
if code == "maxlag":
lag = lagpattern.search(info)
+ pywikibot.log('Pausing due to database lag: ' + info)
if lag:
- pywikibot.log(
- u"Pausing due to database lag: " + info)
- self.site.throttle.lag(int(lag.group("lag")))
- continue
+ lag = lag.group('lag')
+ self.site.throttle.lag(int(lag or 0))
+ continue
+
elif code == 'help' and self.action == 'help':
# The help module returns an error result with the complete
# API information. As this data was requested, return the
diff --git a/pywikibot/throttle.py b/pywikibot/throttle.py
index ebad441..ab56ac7 100644
--- a/pywikibot/throttle.py
+++ b/pywikibot/throttle.py
@@ -65,6 +65,7 @@
# Free the process id after this many seconds:
self.releasepid = 1200
+ self.retry_after = 0 # set by http.request
self.delay = 0
self.checktime = 0
self.multiplydelay = multiplydelay
@@ -274,18 +275,27 @@
else:
self.last_read = time.time()
- def lag(self, lagtime):
+ def lag(self, lagtime=None):
"""Seize the throttle lock due to server lag.
- This will prevent any thread from accessing this site.
+ Usually the self.retry-after value from response_header of the last
+ request if available which will be used for wait time. Default value
+ set by api and stored in self.retry_after is 5. If neither retry_after
+ nor lagtime is set, fallback to 5.
+ This method is used by api.request. It will prevent any thread from
+ accessing this site.
+
+ @param lagtime: The time to wait for the next request which is the
+ last maxlag time from api warning. This is only used as a fallback
+ if self.retry-after isn't set.
+ @type lagtime: int
"""
started = time.time()
with self.lock:
- # start at 1/2 the current server lag time
- # wait at least 5 seconds but not more than 120 seconds
- delay = min(max(5, lagtime // 2), 120)
+ waittime = self.retry_after or lagtime or 5
+ # wait not more than 120 seconds
+ delay = min(waittime, 120)
# account for any time we waited while acquiring the lock
wait = delay - (time.time() - started)
-
self.wait(wait)
diff --git a/tests/api_tests.py b/tests/api_tests.py
index 041af5f..c98b482 100644
--- a/tests/api_tests.py
+++ b/tests/api_tests.py
@@ -1084,9 +1084,10 @@
pywikibot.warning(
'Wrong api.lagpattern regex, cannot retrieve lag value')
raise e
- value = mysite.throttle._lagvalue
- self.assertIsInstance(value, int)
- self.assertGreaterEqual(value, 0)
+ self.assertIsInstance(mythrottle._lagvalue, int)
+ self.assertGreaterEqual(mythrottle._lagvalue, 0)
+ self.assertIsInstance(mythrottle.retry_after, int)
+ self.assertGreaterEqual(mythrottle.retry_after, 0)
def test_individual_patterns(self):
"""Test api.lagpattern with example patterns."""
--
To view, visit https://gerrit.wikimedia.org/r/307127
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie9666ba7654ab071bfeda2be53b29c3c568e508d
Gerrit-Change-Number: 307127
Gerrit-PatchSet: 17
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Multichill <maarten(a)mdammers.nl>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>