jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/817766 )
Change subject: [IMPR] fix typos in logging.py
......................................................................
[IMPR] fix typos in logging.py
Fix typos in docstring.
Change-Id: I747933f96e7f8e54366b2e4dd8066215303055bb
---
M pywikibot/logging.py
1 file changed, 6 insertions(+), 5 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/logging.py b/pywikibot/logging.py
index a955887..e6162c2 100644
--- a/pywikibot/logging.py
+++ b/pywikibot/logging.py
@@ -35,6 +35,7 @@
from pywikibot.backports import Callable, List
from pywikibot.tools import deprecated_args, issue_deprecation_warning
+
STDOUT = 16 #:
VERBOSE = 18 #:
INPUT = 25 #:
@@ -87,7 +88,7 @@
the log message to include an exception traceback.
:param msg: The message to be printed.
- :param args: Not used yet; prevents positinal arguments except `msg`.
+ :param args: Not used yet; prevents positional arguments except `msg`.
:param level: The logging level; supported by :func:`logoutput` only.
:keyword newline: If newline is True (default), a line feed will be
added after printing the msg.
@@ -95,7 +96,7 @@
:keyword layer: Suffix of the logger name separated by dot. By
default no suffix is used.
:type layer: str
- :keyword decoder: If msg is bytes, this decoder is used to deccode.
+ :keyword decoder: If msg is bytes, this decoder is used to decode.
Default is 'utf-8', fallback is 'iso8859-1'
:type decoder: str
:param kwargs: For the other keyword arguments refer
@@ -170,7 +171,7 @@
output = info
-"""Synomym for :func:`info` for backward compatibility. The arguments
+"""Synonym for :func:`info` for backward compatibility. The arguments
are interpreted as for :func:`logoutput`.
.. versionchanged:: 7.2
@@ -189,13 +190,13 @@
``msg`` will be sent to standard output (stdout) via
:mod:`pywikibot.userinterfaces`, so that it can be piped to another
- process. All other functions will sent to stderr.
+ process. All other functions will send to stderr.
`msg` may be omitted and a newline is printed in that case.
The arguments are interpreted as for :func:`logoutput`.
.. versionchanged:: 7.2
- `text`was renamed to `msg`; `msg` paramerer may be omitted;
+ `text`was renamed to `msg`; `msg` parameter may be omitted;
only keyword arguments are allowed except for `msg`.
.. seealso::
- :python:`Logger.log()<library/logging.html#logging.Logger.log>`
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/817766
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I747933f96e7f8e54366b2e4dd8066215303055bb
Gerrit-Change-Number: 817766
Gerrit-PatchSet: 1
Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
Xqt has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/817363 )
Change subject: [backport] backport archivebot.py from master
......................................................................
[backport] backport archivebot.py from master
- Non latin digits support introduced with
https://gerrit.wikimedia.org/r/c/pywikibot/core/+/163213
never worked because the variable replacements like %(counter)d
expected an int instead of s str. This did not fail as long as
textlib.to_local_digits returned an unchanged value if there are no
local digits given for a language but it might be failed for those
who have it. With 7.5 textlib.to_local_digits always return a str
and the archivebot failed. This was fixed recently with 7.5.1.
- User should be able to decide whether to use latin or non latin digits.
Therefore a lot for new fields were introduced like 'localcounter'
which uses the localized number instead of the latin one. This does
not break the further implementation due to the %d replacement
except in rare cases if the user had it replaced by %s already.
- Restore old values for non local fields
- Remove the 7.5.1 changes
- make a sanity check in analyze_page() method for the case that the
local fields are used with %d and show a warning in this case.
- Use a generator instead of a list of pages to process. This decreases
memory usage a lot and also speeds up start time by giving up sorting
all pages.
- preload the page contents
- catch KeyboardInterrupt and leave the mean loop
- add look & feel of CurrentPageBot
- print execution time finally
Bug: T71551
Bug: T313682
Bug: T313692
Bug: T313785
Change-Id: I10c8ac62656aa53f41be629003e5ed6a875f9310
---
M pywikibot/__metadata__.py
M scripts/archivebot.py
2 files changed, 82 insertions(+), 56 deletions(-)
Approvals:
Xqt: Verified; Looks good to me, approved
diff --git a/pywikibot/__metadata__.py b/pywikibot/__metadata__.py
index 21e0e01..e00d7eb 100644
--- a/pywikibot/__metadata__.py
+++ b/pywikibot/__metadata__.py
@@ -11,7 +11,7 @@
__name__ = 'pywikibot'
-__version__ = '7.5.1'
+__version__ = '7.5.2'
__description__ = 'Python MediaWiki Bot Framework'
__maintainer__ = 'The Pywikibot team'
__maintainer_email__ = 'pywikibot(a)lists.wikimedia.org'
diff --git a/scripts/archivebot.py b/scripts/archivebot.py
index ecc33d6..33c776e 100755
--- a/scripts/archivebot.py
+++ b/scripts/archivebot.py
@@ -52,18 +52,33 @@
key A secret key that (if valid) allows archives not to be
subpages of the page being archived.
-Variables below can be used in the value for "archive" in the template above:
+Variables below can be used in the value for "archive" in the template
+above; numbers are latin digits:
-%(counter)s the current value of the counter
-%(year)s year of the thread being archived
-%(isoyear)s ISO year of the thread being archived
-%(isoweek)s ISO week number of the thread being archived
-%(semester)s semester term of the year of the thread being archived
-%(quarter)s quarter of the year of the thread being archived
-%(month)s month (as a number 1-12) of the thread being archived
+%(counter)d the current value of the counter
+%(year)d year of the thread being archived
+%(isoyear)d ISO year of the thread being archived
+%(isoweek)d ISO week number of the thread being archived
+%(semester)d semester term of the year of the thread being archived
+%(quarter)d quarter of the year of the thread being archived
+%(month)d month (as a number 1-12) of the thread being archived
%(monthname)s localized name of the month above
%(monthnameshort)s first three letters of the name above
-%(week)s week number of the thread being archived
+%(week)d week number of the thread being archived
+
+Alternatively you may use localized digits. This is only available for a
+few site languages. Refer :attr:`NON_LATIN_DIGITS
+<pywikibot.userinterfaces.transliteration.NON_LATIN_DIGITS>` whether
+there is a localized one:
+
+%(localcounter)s the current value of the counter
+%(localyear)s year of the thread being archived
+%(localisoyear)s ISO year of the thread being archived
+%(localisoweek)s ISO week number of the thread being archived
+%(localsemester)s semester term of the year of the thread being archived
+%(localquarter)s quarter of the year of the thread being archived
+%(localmonth)s month (as a number 1-12) of the thread being archived
+%(localweek)s week number of the thread being archived
The ISO calendar starts with the Monday of the week which has at least four
days in the new Gregorian calendar. If January 1st is between Monday and
@@ -87,9 +102,8 @@
-page:PAGE archive a single PAGE, default ns is a user talk page
-salt:SALT specify salt
-.. versionchanged:: 7.5.1
- string presentation type should be used for "archive" variable in the
- template to support non latin values
+.. versionchanged:: 7.5.2
+ Localized variables for "archive" template parameter are supported
"""
#
# (C) Pywikibot team, 2006-2022
@@ -104,6 +118,7 @@
from collections import OrderedDict, defaultdict
from hashlib import md5
from math import ceil
+from textwrap import fill
from typing import Any, Optional, Pattern
from warnings import warn
@@ -484,16 +499,10 @@
return self.get_attr('key') == hexdigest
def load_config(self) -> None:
- """Load and validate archiver template.
-
- .. versionchanged:: 7.5.1
- replace archive pattern fields to string conversion
- """
+ """Load and validate archiver template."""
pywikibot.info('Looking for: {{{{{}}}}} in {}'
.format(self.tpl.title(), self.page))
- fields = self.get_params(self.now, 0).keys() # dummy parameters
- pattern = re.compile(r'%(\((?:{})\))d'.format('|'.join(fields)))
for tpl, params in self.page.raw_extracted_templates:
try: # Check tpl name before comparing; it might be invalid.
tpl_page = pywikibot.Page(self.site, tpl, ns=10)
@@ -503,11 +512,7 @@
if tpl_page == self.tpl:
for item, value in params.items():
- # convert archive pattern fields to string
- # to support non latin digits
- if item == 'archive':
- value = pattern.sub(r'%\1s', value)
- self.set_attr(item.strip(), value.strip())
+ self.set_attr(item, value)
break
else:
raise MissingConfigError('Missing or malformed template')
@@ -562,20 +567,22 @@
def get_params(self, timestamp, counter: int) -> dict:
"""Make params for archiving template."""
lang = self.site.lang
- return {
- 'counter': to_local_digits(counter, lang),
- 'year': to_local_digits(timestamp.year, lang),
- 'isoyear': to_local_digits(timestamp.isocalendar()[0], lang),
- 'isoweek': to_local_digits(timestamp.isocalendar()[1], lang),
- 'semester': to_local_digits(int(ceil(timestamp.month / 6)), lang),
- 'quarter': to_local_digits(int(ceil(timestamp.month / 3)), lang),
- 'month': to_local_digits(timestamp.month, lang),
- 'monthname': self.month_num2orig_names[timestamp.month]['long'],
- 'monthnameshort': self.month_num2orig_names[
- timestamp.month]['short'],
- 'week': to_local_digits(
- int(time.strftime('%W', timestamp.timetuple())), lang),
+ params = {
+ 'counter': counter,
+ 'year': timestamp.year,
+ 'isoyear': timestamp.isocalendar()[0],
+ 'isoweek': timestamp.isocalendar()[1],
+ 'semester': int(ceil(timestamp.month / 6)),
+ 'quarter': int(ceil(timestamp.month / 3)),
+ 'month': timestamp.month,
+ 'week': int(time.strftime('%W', timestamp.timetuple())),
}
+ params.update({'local' + key: to_local_digits(value, lang)
+ for key, value in params.items()})
+ monthnames = self.month_num2orig_names[timestamp.month]
+ params['monthname'] = monthnames['long']
+ params['monthnameshort'] = monthnames['short']
+ return params
def analyze_page(self) -> Set[ShouldArchive]:
"""Analyze DiscussionPage."""
@@ -588,6 +595,9 @@
whys = set()
pywikibot.output('Processing {} threads'
.format(len(self.page.threads)))
+ fields = self.get_params(self.now, 0).keys() # dummy parameters
+ regex = re.compile(r'%(\((?:{})\))d'.format('|'.join(fields)))
+ stringpattern = regex.sub(r'%\1s', pattern)
for i, thread in enumerate(self.page.threads):
# TODO: Make an option so that unstamped (unsigned) posts get
# archived.
@@ -598,7 +608,21 @@
params = self.get_params(thread.timestamp, counter)
# this is actually just a dummy key to group the threads by
# "era" regardless of the counter and deal with it later
- key = pattern % params
+ try:
+ key = pattern % params
+ except TypeError as e:
+ if 'a real number is required' in str(e):
+ pywikibot.error(e)
+ pywikibot.info(
+ fill('<<lightblue>>Use string format field like '
+ '%(localfield)s instead of %(localfield)d. '
+ 'Trying to solve it...'))
+ pywikibot.info()
+ pattern = stringpattern
+ key = pattern % params
+ else:
+ raise MalformedConfigError(e)
+
threads_per_archive[key].append((i, thread))
whys.add(why) # xxx: we don't know if we ever archive anything
@@ -791,26 +815,22 @@
return
for template_name in templates:
- pagelist = []
tmpl = pywikibot.Page(site, template_name, ns=10)
- if not filename and not pagename:
- if namespace is not None:
- ns = [str(namespace)]
- else:
- ns = []
- pywikibot.output('Fetching template transclusions...')
- pagelist.extend(tmpl.getReferences(only_template_inclusion=True,
- follow_redirects=False,
- namespaces=ns))
if filename:
with open(filename) as f:
- for pg in f.readlines():
- pagelist.append(pywikibot.Page(site, pg, ns=10))
- if pagename:
- pagelist.append(pywikibot.Page(site, pagename, ns=3))
- pagelist.sort()
- for pg in pagelist:
- pywikibot.output('Processing {}'.format(pg))
+ gen = [pywikibot.Page(site, line, ns=10) for line in f]
+ elif pagename:
+ gen = [pywikibot.Page(site, pagename, ns=3)]
+ else:
+ ns = [str(namespace)] if namespace is not None else []
+ pywikibot.output('Fetching template transclusions...')
+ gen = tmpl.getReferences(only_template_inclusion=True,
+ follow_redirects=False,
+ namespaces=ns,
+ content=True)
+ for pg in gen:
+ pywikibot.info('\n\n>>> <<lightpurple>>{}<<default>> <<<'
+ .format(pg.title()))
# Catching exceptions, so that errors in one page do not bail out
# the entire process
try:
@@ -823,7 +843,13 @@
except Exception:
pywikibot.exception('Error occurred while processing page {}'
.format(pg))
+ except KeyboardInterrupt:
+ pywikibot.info('\nUser quit bot run...')
+ return
if __name__ == '__main__':
+ start = datetime.datetime.now()
main()
+ pywikibot.info('\nExecution time: {} seconds'
+ .format((datetime.datetime.now() - start).seconds))
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/817363
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: stable
Gerrit-Change-Id: I10c8ac62656aa53f41be629003e5ed6a875f9310
Gerrit-Change-Number: 817363
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/816832 )
Change subject: [IMPR] decrease memory usage and improve processing speed
......................................................................
[IMPR] decrease memory usage and improve processing speed
- Use a generator instead of a list of pages to process. This decreases
memory usage a lot and also speeds up start time by giving up sorting
all pages.
- preload the page contents
- catch KeyboardInterrupt and leave the mean loop
- add look & feel of CurrentPageBot
- print execution time finally
Change-Id: Ib3572e1cf76ce898bc238b6d40688c286603cdd9
---
M scripts/archivebot.py
1 file changed, 19 insertions(+), 17 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/archivebot.py b/scripts/archivebot.py
index e92448c..19ab8cf 100755
--- a/scripts/archivebot.py
+++ b/scripts/archivebot.py
@@ -815,26 +815,22 @@
return
for template_name in templates:
- pagelist = []
tmpl = pywikibot.Page(site, template_name, ns=10)
- if not filename and not pagename:
- if namespace is not None:
- ns = [str(namespace)]
- else:
- ns = []
- pywikibot.output('Fetching template transclusions...')
- pagelist.extend(tmpl.getReferences(only_template_inclusion=True,
- follow_redirects=False,
- namespaces=ns))
if filename:
with open(filename) as f:
- for pg in f.readlines():
- pagelist.append(pywikibot.Page(site, pg, ns=10))
- if pagename:
- pagelist.append(pywikibot.Page(site, pagename, ns=3))
- pagelist.sort()
- for pg in pagelist:
- pywikibot.output('Processing {}'.format(pg))
+ gen = [pywikibot.Page(site, line, ns=10) for line in f]
+ elif pagename:
+ gen = [pywikibot.Page(site, pagename, ns=3)]
+ else:
+ ns = [str(namespace)] if namespace is not None else []
+ pywikibot.output('Fetching template transclusions...')
+ gen = tmpl.getReferences(only_template_inclusion=True,
+ follow_redirects=False,
+ namespaces=ns,
+ content=True)
+ for pg in gen:
+ pywikibot.info('\n\n>>> <<lightpurple>>{}<<default>> <<<'
+ .format(pg.title()))
# Catching exceptions, so that errors in one page do not bail out
# the entire process
try:
@@ -847,7 +843,13 @@
except Exception:
pywikibot.exception('Error occurred while processing page {}'
.format(pg))
+ except KeyboardInterrupt:
+ pywikibot.info('\nUser quit bot run...')
+ return
if __name__ == '__main__':
+ start = datetime.datetime.now()
main()
+ pywikibot.info('\nExecution time: {} seconds'
+ .format((datetime.datetime.now() - start).seconds))
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/816832
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Ib3572e1cf76ce898bc238b6d40688c286603cdd9
Gerrit-Change-Number: 816832
Gerrit-PatchSet: 5
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com>
Gerrit-Reviewer: PotsdamLamb
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged