jenkins-bot has submitted this change and it was merged.
Change subject: Safer use of mementos returned data structure
......................................................................
Safer use of mementos returned data structure
Bug: T112250
Change-Id: Ia36e0e88142c59a7f625b90c1be0b8f5a3c26614
---
M scripts/weblinkchecker.py
1 file changed, 13 insertions(+), 1 deletion(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/weblinkchecker.py b/scripts/weblinkchecker.py
index 64a01a3..a918601 100755
--- a/scripts/weblinkchecker.py
+++ b/scripts/weblinkchecker.py
@@ -178,7 +178,19 @@
mc.timegate_uri = timegate_uri
memento_info = mc.get_memento_info(url, when)
- return memento_info.get('mementos').get('closest').get('uri')[0]
+ mementos = memento_info.get('mementos')
+ if not mementos:
+ raise Exception(
+ 'mementos not found for {0} via {1}'.format(url, timegate_uri))
+ if 'closest' not in mementos:
+ raise Exception(
+ 'closest memento not found for {0} via {1}'.format(
+ url, timegate_uri))
+ if 'uri' not in mementos['closest']:
+ raise Exception(
+ 'closest memento uri not found for {0} via {1}'.format(
+ url, timegate_uri))
+ return mementos['closest']['uri'][0]
def get_archive_url(url):
--
To view, visit https://gerrit.wikimedia.org/r/238999
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia36e0e88142c59a7f625b90c1be0b8f5a3c26614
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [FIX] bot: Fix the module documentation
......................................................................
[FIX] bot: Fix the module documentation
The documentation added in e8170759 didn't really like the lists.
Change-Id: Ie1df833a4056ac6a6f762cc20dcbd8ad225c9dca
---
M pywikibot/bot.py
1 file changed, 13 insertions(+), 10 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/bot.py b/pywikibot/bot.py
index dae9760..aa4b41d 100644
--- a/pywikibot/bot.py
+++ b/pywikibot/bot.py
@@ -6,18 +6,21 @@
conjunction. Each bot should subclass at least one of these four classes:
* L{BaseBot}: Basic bot class in case where the site is handled differently,
- like working on two sites in parallel.
+ like working on two sites in parallel.
+
* L{SingleSiteBot}: Bot class which should only be run on a single site. They
- usually store site specific content and thus can't be easily run when the
- generator returns a page on another site. It has a property C{site} which
- can also be changed. If the generator returns a page of a different site
- it'll skip that page.
+ usually store site specific content and thus can't be easily run when the
+ generator returns a page on another site. It has a property C{site} which
+ can also be changed. If the generator returns a page of a different site
+ it'll skip that page.
+
* L{MultipleSitesBot}: Bot class which supports to be run on multiple sites
- without the need to manually initialize it every time. It is not possible to
- set the C{site} property and it's deprecated to request it. Instead site of
- the current page should be used. And out of C{run} that sit isn't defined.
+ without the need to manually initialize it every time. It is not possible to
+ set the C{site} property and it's deprecated to request it. Instead site of
+ the current page should be used. And out of C{run} that sit isn't defined.
+
* L{Bot}: The previous base class which should be avoided. This class is mainly
- used for bots which work with wikibase or together with an image repository.
+ used for bots which work with wikibase or together with an image repository.
Additionally there is the L{CurrentPageBot} class which automatically sets the
current page to the page treated. It is recommended to use this class and to
@@ -32,7 +35,7 @@
* L{RedirectPageBot}: Only handle pages which are redirect pages.
* L{NoRedirectPageBot}: Only handle pages which are not redirect pages.
* L{FollowRedirectPageBot}: If the generator returns a redirect page it'll
- follow the redirect and instead work on the redirected class.
+ follow the redirect and instead work on the redirected class.
It is possible to combine filters by subclassing multiple of them. They are
new-style classes so when a class is first subclassing L{ExistingPageBot} and
--
To view, visit https://gerrit.wikimedia.org/r/237994
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie1df833a4056ac6a6f762cc20dcbd8ad225c9dca
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Move logging to new pywikibot.logging module
......................................................................
Move logging to new pywikibot.logging module
config2 and tools now use logging layer.
pywikibot.__url__ introduced as base url to
online documentation.
Generic logging tools moved into tools._logging
Change-Id: I8f1e2c1d09b5d39c5facadb0690ee71bdbf5de1b
---
M pywikibot/__init__.py
M pywikibot/backports.py
M pywikibot/bot.py
M pywikibot/config2.py
M pywikibot/family.py
M pywikibot/flow.py
M pywikibot/i18n.py
A pywikibot/logging.py
M pywikibot/page.py
M pywikibot/tools/__init__.py
A pywikibot/tools/_logging.py
M pywikibot/userinterfaces/gui.py
M pywikibot/userinterfaces/terminal_interface_base.py
M tox.ini
14 files changed, 393 insertions(+), 314 deletions(-)
Approvals:
XZise: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py
index febe973..fc32f30 100644
--- a/pywikibot/__init__.py
+++ b/pywikibot/__init__.py
@@ -9,6 +9,7 @@
__release__ = '2.0b3'
__version__ = '$Id$'
+__url__ = 'https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Pywikibot'
import datetime
import math
diff --git a/pywikibot/backports.py b/pywikibot/backports.py
index cb44b6e..39b904c 100644
--- a/pywikibot/backports.py
+++ b/pywikibot/backports.py
@@ -12,7 +12,7 @@
# Distributed under the terms of the PSF license.
#
-from __future__ import unicode_literals
+from __future__ import absolute_import, unicode_literals
__license__ = """
PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
@@ -62,6 +62,7 @@
agrees to be bound by the terms and conditions of this License
Agreement.
"""
+
import logging
import warnings
diff --git a/pywikibot/bot.py b/pywikibot/bot.py
index 104a45a..63be6b0 100644
--- a/pywikibot/bot.py
+++ b/pywikibot/bot.py
@@ -49,7 +49,7 @@
#
# Distributed under the terms of the MIT license.
#
-from __future__ import unicode_literals
+from __future__ import absolute_import, unicode_literals
__version__ = '$Id$'
@@ -75,12 +75,6 @@
_logger = "bot"
-# logging levels
-from logging import DEBUG, INFO, WARNING, ERROR, CRITICAL
-STDOUT = 16
-VERBOSE = 18
-INPUT = 25
-
import pywikibot
from pywikibot import backports
@@ -92,7 +86,18 @@
ListOption, HighlightContextOption,
ChoiceException, QuitKeyboardInterrupt,
)
+from pywikibot.logging import CRITICAL, ERROR, INFO, WARNING # noqa: unused
+from pywikibot.logging import DEBUG, INPUT, STDOUT, VERBOSE
+from pywikibot.logging import (
+ add_init_routine,
+ debug, error, exception, log, output, stdout, warning,
+)
+from pywikibot.logging import critical # noqa: unused
from pywikibot.tools import deprecated, deprecated_args, PY2, PYTHON_VERSION
+from pywikibot.tools._logging import (
+ LoggingFormatter as _LoggingFormatter,
+ RotatingFileHandler,
+)
if not PY2:
unicode = str
@@ -130,119 +135,13 @@
self.stop = stop
-# Logging module configuration
-class RotatingFileHandler(logging.handlers.RotatingFileHandler):
+class LoggingFormatter(_LoggingFormatter):
- """Modified RotatingFileHandler supporting unlimited amount of backups."""
+ """Logging formatter that uses config.console_encoding."""
- def doRollover(self):
- """
- Modified naming system for logging files.
-
- Overwrites the default Rollover renaming by inserting the count number
- between file name root and extension. If backupCount is >= 1, the system
- will successively create new files with the same pathname as the base
- file, but with inserting ".1", ".2" etc. in front of the filename
- suffix. For example, with a backupCount of 5 and a base file name of
- "app.log", you would get "app.log", "app.1.log", "app.2.log", ...
- through to "app.5.log". The file being written to is always "app.log" -
- when it gets filled up, it is closed and renamed to "app.1.log", and if
- files "app.1.log", "app.2.log" etc. already exist, then they are
- renamed to "app.2.log", "app.3.log" etc. respectively.
- If backupCount is == -1 do not rotate but create new numbered filenames.
- The newest file has the highest number except some older numbered files
- where deleted and the bot was restarted. In this case the ordering
- starts from the lowest available (unused) number.
-
- """
- if self.stream:
- self.stream.close()
- self.stream = None
- root, ext = os.path.splitext(self.baseFilename)
- if self.backupCount > 0:
- for i in range(self.backupCount - 1, 0, -1):
- sfn = "%s.%d%s" % (root, i, ext)
- dfn = "%s.%d%s" % (root, i + 1, ext)
- if os.path.exists(sfn):
- if os.path.exists(dfn):
- os.remove(dfn)
- os.rename(sfn, dfn)
- dfn = "%s.1%s" % (root, ext)
- if os.path.exists(dfn):
- os.remove(dfn)
- os.rename(self.baseFilename, dfn)
- elif self.backupCount == -1:
- if not hasattr(self, '_lastNo'):
- self._lastNo = 1
- while True:
- fn = "%s.%d%s" % (root, self._lastNo, ext)
- self._lastNo += 1
- if not os.path.exists(fn):
- break
- os.rename(self.baseFilename, fn)
- self.mode = 'w'
- self.stream = self._open()
-
- def format(self, record):
- """Strip trailing newlines before outputting text to file."""
- # Warnings captured from the warnings system are not processed by
- # logoutput(), so the 'context' variables are missing.
- # The same context details are provided by Python 3.X, but need to
- # be extracted from the warning message for Python <= 2.7.
- if record.name == 'py.warnings' and 'caller_file' not in record.__dict__:
- assert len(record.args) == 1, \
- 'Arguments for record is not correctly set'
- msg = record.args[0]
-
- if PY2:
- record.pathname = msg.partition(':')[0]
- record.lineno = msg.partition(':')[2].partition(':')[0]
- record.module = msg.rpartition('/')[2].rpartition('.')[0]
- else:
- assert msg.startswith(record.pathname + ':'), \
- 'Record argument should start with path'
-
- record.__dict__['caller_file'] = record.pathname
- record.__dict__['caller_name'] = record.module
- record.__dict__['caller_line'] = record.lineno
-
- # Remove the path and the line number, and strip the extra space
- msg = msg.partition(':')[2].partition(':')[2].lstrip()
- record.args = (msg,)
-
- text = logging.handlers.RotatingFileHandler.format(self, record)
- return text.rstrip("\r\n")
-
-
-class LoggingFormatter(logging.Formatter):
-
- """Format LogRecords for output to file.
-
- This formatter *ignores* the 'newline' key of the LogRecord, because
- every record written to a file must end with a newline, regardless of
- whether the output to the user's console does.
-
- """
-
- def formatException(self, ei):
- r"""
- Convert exception trace to unicode if necessary.
-
- Make sure that the exception trace is converted to unicode.
-
- L{exceptions.Error} traces are encoded in our console encoding, which
- is needed for plainly printing them. However, when logging them
- using logging.exception, the Python logging module will try to use
- these traces, and it will fail if they are console encoded strings.
-
- Formatter.formatException also strips the trailing \n, which we need.
- """
- strExc = logging.Formatter.formatException(self, ei)
-
- if PY2 and isinstance(strExc, bytes):
- return strExc.decode(config.console_encoding) + '\n'
- else:
- return strExc + '\n'
+ def __init__(self, fmt=None, datefmt=None):
+ """Constructor setting underlying encoding to console_encoding."""
+ _LoggingFormatter.__init__(self, fmt, datefmt, config.console_encoding)
# Initialize the handlers and formatters for the logging system.
@@ -456,162 +355,7 @@
log(u'=== ' * 14)
-# User output/logging functions
-
-# Six output functions are defined. Each requires a unicode or string
-# argument. All of these functions generate a message to the log file if
-# logging is enabled ("-log" or "-debug" command line arguments).
-
-# The functions output(), stdout(), warning(), and error() all display a
-# message to the user through the logger object; the only difference is the
-# priority level, which can be used by the application layer to alter the
-# display. The stdout() function should be used only for data that is
-# the "result" of a script, as opposed to information messages to the
-# user.
-
-# The function log() by default does not display a message to the user, but
-# this can be altered by using the "-verbose" command line option.
-
-# The function debug() only logs its messages, they are never displayed on
-# the user console. debug() takes a required second argument, which is a
-# string indicating the debugging layer.
-
-def logoutput(text, decoder=None, newline=True, _level=INFO, _logger="",
- **kwargs):
- """Format output and send to the logging module.
-
- Helper function used by all the user-output convenience functions.
-
- """
- if _logger:
- logger = logging.getLogger("pywiki." + _logger)
- else:
- logger = logging.getLogger("pywiki")
-
- # make sure logging system has been initialized
- if not _handlers_initialized:
- init_handlers()
-
- # frame 0 is logoutput() in this module,
- # frame 1 is the convenience function (output(), etc.)
- # frame 2 is whatever called the convenience function
- frame = sys._getframe(2)
-
- module = os.path.basename(frame.f_code.co_filename)
- context = {'caller_name': frame.f_code.co_name,
- 'caller_file': module,
- 'caller_line': frame.f_lineno,
- 'newline': ("\n" if newline else "")}
-
- if decoder:
- text = text.decode(decoder)
- elif not isinstance(text, unicode):
- if not isinstance(text, str):
- # looks like text is a non-text object.
- # Maybe it has a __unicode__ builtin ?
- # (allows to print Page, Site...)
- text = unicode(text)
- else:
- try:
- text = text.decode('utf-8')
- except UnicodeDecodeError:
- text = text.decode('iso8859-1')
-
- logger.log(_level, text, extra=context, **kwargs)
-
-
-def output(text, decoder=None, newline=True, toStdout=False, **kwargs):
- r"""Output a message to the user via the userinterface.
-
- Works like print, but uses the encoding used by the user's console
- (console_encoding in the configuration file) instead of ASCII.
-
- If decoder is None, text should be a unicode string. Otherwise it
- should be encoded in the given encoding.
-
- If newline is True, a line feed will be added after printing the text.
-
- If toStdout is True, the text will be sent to standard output,
- so that it can be piped to another process. All other text will
- be sent to stderr. See: https://en.wikipedia.org/wiki/Pipeline_%28Unix%29
-
- text can contain special sequences to create colored output. These
- consist of the escape character \03 and the color name in curly braces,
- e. g. \03{lightpurple}. \03{default} resets the color.
-
- Other keyword arguments are passed unchanged to the logger; so far, the
- only argument that is useful is "exc_info=True", which causes the
- log message to include an exception traceback.
-
- """
- if toStdout: # maintained for backwards-compatibity only
- logoutput(text, decoder, newline, STDOUT, **kwargs)
- else:
- logoutput(text, decoder, newline, INFO, **kwargs)
-
-
-def stdout(text, decoder=None, newline=True, **kwargs):
- """Output script results to the user via the userinterface."""
- logoutput(text, decoder, newline, STDOUT, **kwargs)
-
-
-def warning(text, decoder=None, newline=True, **kwargs):
- """Output a warning message to the user via the userinterface."""
- logoutput(text, decoder, newline, WARNING, **kwargs)
-
-
-def error(text, decoder=None, newline=True, **kwargs):
- """Output an error message to the user via the userinterface."""
- logoutput(text, decoder, newline, ERROR, **kwargs)
-
-
-def log(text, decoder=None, newline=True, **kwargs):
- """Output a record to the log file."""
- logoutput(text, decoder, newline, VERBOSE, **kwargs)
-
-
-def critical(text, decoder=None, newline=True, **kwargs):
- """Output a critical record to the log file."""
- logoutput(text, decoder, newline, CRITICAL, **kwargs)
-
-
-def debug(text, layer, decoder=None, newline=True, **kwargs):
- """Output a debug record to the log file.
-
- @param layer: The name of the logger that text will be sent to.
- """
- logoutput(text, decoder, newline, DEBUG, layer, **kwargs)
-
-
-def exception(msg=None, decoder=None, newline=True, tb=False, **kwargs):
- """Output an error traceback to the user via the userinterface.
-
- Use directly after an 'except' statement::
-
- ...
- except:
- pywikibot.exception()
- ...
-
- or alternatively::
-
- ...
- except Exception as e:
- pywikibot.exception(e)
- ...
-
- @param tb: Set to True in order to output traceback also.
- """
- if isinstance(msg, BaseException):
- exc_info = 1
- else:
- exc_info = sys.exc_info()
- msg = u'%s: %s' % (repr(exc_info[1]).split('(')[0],
- unicode(exc_info[1]).strip())
- if tb:
- kwargs['exc_info'] = exc_info
- logoutput(msg, decoder, newline, ERROR, **kwargs)
-
+add_init_routine(init_handlers)
# User input functions
diff --git a/pywikibot/config2.py b/pywikibot/config2.py
index 774a818..a19def2 100644
--- a/pywikibot/config2.py
+++ b/pywikibot/config2.py
@@ -56,6 +56,9 @@
else:
import _winreg as winreg
+from pywikibot import __url__
+from pywikibot.logging import error, output, warning
+
# This frozen set should contain all imported modules/variables, so it must
# occur directly after the imports. At that point globals() only contains the
# names and some magic variables (like __name__)
@@ -317,7 +320,7 @@
exc_text = "No user-config.py found in directory '%s'.\n" % base_dir
if _no_user_config:
if _no_user_config != '2':
- print(exc_text)
+ output(exc_text)
else:
exc_text += " Please check that user-config.py is stored in the correct location.\n"
exc_text += " Directory where user-config.py is searched is determined as follows:\n\n"
@@ -333,7 +336,7 @@
for arg in sys.argv[1:]:
if arg.startswith(str('-verbose')) or arg == str('-v'):
- print("The base directory is %s" % base_dir)
+ output('The base directory is {0}'.format(base_dir))
break
family_files = {}
@@ -871,8 +874,8 @@
return cmd[:-1].strip()
except WindowsError as e:
# Catch any key lookup errors
- print('WARNING: Unable to find editor for files *.' + extension)
- print(e)
+ output('Unable to detect program for file extension "{0}": {1}'.format(
+ extension, e))
def _detect_win32_editor():
@@ -922,7 +925,7 @@
_thislevel = 0
if _no_user_config:
if _no_user_config != '2':
- print("WARNING: Skipping loading of user-config.py.")
+ warning('Skipping loading of user-config.py.')
_fns = []
else:
_fns = [os.path.join(_base_dir, "user-config.py")]
@@ -937,11 +940,11 @@
with open(_filename, 'rb') as f:
exec(compile(f.read(), _filename, 'exec'), _uc)
else:
- print("WARNING: Skipped '%(fn)s': writeable by others."
- % {'fn': _filename})
+ warning("Skipped '%(fn)s': writeable by others."
+ % {'fn': _filename})
else:
- print("WARNING: Skipped '%(fn)s': owned by someone else."
- % {'fn': _filename})
+ warning("Skipped '%(fn)s': owned by someone else."
+ % {'fn': _filename})
class _DifferentTypeError(UserWarning, TypeError):
@@ -1031,10 +1034,9 @@
if transliteration_target == 'not set':
if sys.platform == 'win32':
transliteration_target = console_encoding
- print("WARNING: Running on Windows and transliteration_target is not "
- "set.")
- print("Please see "
- "https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Pywikibot/Windows")
+ warning(
+ 'Running on Windows and transliteration_target is not set.\n'
+ 'Please see {0}/Windows'.format(__url__))
else:
transliteration_target = None
elif transliteration_target in ('None', 'none'):
@@ -1049,23 +1051,24 @@
# https://docs.python.org/2/reference/lexical_analysis.html#string-literals
# encode('unicode-escape') also changes Unicode characters
if set(editor) & set('\a\b\f\n\r\t\v'):
- print('WARNING: The editor path contains probably invalid escaped '
- 'characters. Make sure to use a raw-string (r"..." or r\'...\'), '
- 'forward slashs as a path delimiter or to escape the normal '
- 'path delimiter.')
+ warning(
+ 'The editor path contains probably invalid escaped '
+ 'characters. Make sure to use a raw-string (r"..." or r\'...\'), '
+ 'forward slashs as a path delimiter or to escape the normal '
+ 'path delimiter.')
# Fix up default site
if family == 'wikipedia' and mylang == 'language':
if _no_user_config != '2':
- print("WARNING: family and mylang are not set.\n"
- "Defaulting to family='test' and mylang='test'.")
+ warning('family and mylang are not set.\n'
+ "Defaulting to family='test' and mylang='test'.")
family = mylang = 'test'
# SECURITY WARNINGS
if (not ignore_file_security_warnings and
private_files_permission & (stat.S_IRWXG | stat.S_IRWXO) != 0):
- print("CRITICAL SECURITY WARNING: 'private_files_permission' is set"
+ error("CRITICAL SECURITY WARNING: 'private_files_permission' is set"
" to allow access from the group/others which"
" could give them access to the sensitive files."
" To avoid giving others access to sensitive files, pywikibot"
@@ -1083,7 +1086,7 @@
if _arg == "modified":
_all = 0
else:
- print("Unknown arg %(_arg)s ignored" % locals())
+ warning('Unknown arg %(_arg)s ignored' % locals())
_k = list(globals().keys())
_k.sort()
for _name in _k:
@@ -1102,7 +1105,7 @@
_value = repr('xxxxxxxx')
else:
_value = repr(_value)
- print("%s=%s" % (_name, _value))
+ output('{0}={1}'.format(_name, _value))
# cleanup all locally-defined variables
for __var in list(globals().keys()):
diff --git a/pywikibot/family.py b/pywikibot/family.py
index 148a539..72753f0 100644
--- a/pywikibot/family.py
+++ b/pywikibot/family.py
@@ -5,7 +5,7 @@
#
# Distributed under the terms of the MIT license.
#
-from __future__ import unicode_literals
+from __future__ import absolute_import, unicode_literals
__version__ = '$Id$'
#
diff --git a/pywikibot/flow.py b/pywikibot/flow.py
index 46dfac9..accc5d4 100644
--- a/pywikibot/flow.py
+++ b/pywikibot/flow.py
@@ -5,7 +5,7 @@
#
# Distributed under the terms of the MIT license.
#
-from __future__ import unicode_literals
+from __future__ import absolute_import, unicode_literals
__version__ = '$Id$'
diff --git a/pywikibot/i18n.py b/pywikibot/i18n.py
index c8c516c..0a2a5d2 100644
--- a/pywikibot/i18n.py
+++ b/pywikibot/i18n.py
@@ -37,6 +37,7 @@
import pywikibot
+from pywikibot import __url__
from pywikibot import Error
from pywikibot import config
from pywikibot.plural import plural_rules
@@ -477,8 +478,8 @@
raise TranslationError(
'Unable to load messages package %s for bundle %s'
'\nIt can happen due to lack of i18n submodule or files. '
- 'Read https://mediawiki.org/wiki/PWB/i18n'
- % (_messages_package_name, twtitle))
+ 'Read %s/i18n'
+ % (_messages_package_name, twtitle, __url__))
code_needed = False
# If a site is given instead of a code, use its language
diff --git a/pywikibot/logging.py b/pywikibot/logging.py
new file mode 100644
index 0000000..ed82a98
--- /dev/null
+++ b/pywikibot/logging.py
@@ -0,0 +1,200 @@
+# -*- coding: utf-8 -*-
+"""Logging functions."""
+#
+# (C) Pywikibot team, 2010-2015
+#
+# Distributed under the terms of the MIT license.
+#
+from __future__ import absolute_import, unicode_literals
+
+__version__ = '$Id$'
+
+import logging
+import os
+import sys
+
+# logging levels
+from logging import DEBUG, INFO, WARNING, ERROR, CRITICAL
+STDOUT = 16
+VERBOSE = 18
+INPUT = 25
+
+if sys.version_info[0] > 2:
+ unicode = str
+
+_init_routines = []
+_inited_routines = []
+
+
+def add_init_routine(routine):
+ """Add a routine to be run as soon as possible."""
+ _init_routines.append(routine)
+
+
+def _init():
+ """Init any routines which have not already been called."""
+ for init_routine in _init_routines:
+ if init_routine not in _inited_routines:
+ init_routine()
+ _inited_routines.append(init_routine)
+
+ # Clear the list of routines to be inited
+ _init_routines[:] = []
+
+
+# User output/logging functions
+
+# Six output functions are defined. Each requires a unicode or string
+# argument. All of these functions generate a message to the log file if
+# logging is enabled ("-log" or "-debug" command line arguments).
+
+# The functions output(), stdout(), warning(), and error() all display a
+# message to the user through the logger object; the only difference is the
+# priority level, which can be used by the application layer to alter the
+# display. The stdout() function should be used only for data that is
+# the "result" of a script, as opposed to information messages to the
+# user.
+
+# The function log() by default does not display a message to the user, but
+# this can be altered by using the "-verbose" command line option.
+
+# The function debug() only logs its messages, they are never displayed on
+# the user console. debug() takes a required second argument, which is a
+# string indicating the debugging layer.
+
+
+def logoutput(text, decoder=None, newline=True, _level=INFO, _logger="",
+ **kwargs):
+ """Format output and send to the logging module.
+
+ Helper function used by all the user-output convenience functions.
+
+ """
+ if _logger:
+ logger = logging.getLogger("pywiki." + _logger)
+ else:
+ logger = logging.getLogger("pywiki")
+
+ # invoke any init routines
+ if _init_routines:
+ _init()
+
+ # frame 0 is logoutput() in this module,
+ # frame 1 is the convenience function (output(), etc.)
+ # frame 2 is whatever called the convenience function
+ frame = sys._getframe(2)
+
+ module = os.path.basename(frame.f_code.co_filename)
+ context = {'caller_name': frame.f_code.co_name,
+ 'caller_file': module,
+ 'caller_line': frame.f_lineno,
+ 'newline': ("\n" if newline else "")}
+
+ if decoder:
+ text = text.decode(decoder)
+ elif not isinstance(text, unicode):
+ if not isinstance(text, str):
+ # looks like text is a non-text object.
+ # Maybe it has a __unicode__ builtin ?
+ # (allows to print Page, Site...)
+ text = unicode(text)
+ else:
+ try:
+ text = text.decode('utf-8')
+ except UnicodeDecodeError:
+ text = text.decode('iso8859-1')
+
+ logger.log(_level, text, extra=context, **kwargs)
+
+
+def output(text, decoder=None, newline=True, toStdout=False, **kwargs):
+ r"""Output a message to the user via the userinterface.
+
+ Works like print, but uses the encoding used by the user's console
+ (console_encoding in the configuration file) instead of ASCII.
+
+ If decoder is None, text should be a unicode string. Otherwise it
+ should be encoded in the given encoding.
+
+ If newline is True, a line feed will be added after printing the text.
+
+ If toStdout is True, the text will be sent to standard output,
+ so that it can be piped to another process. All other text will
+ be sent to stderr. See: https://en.wikipedia.org/wiki/Pipeline_%28Unix%29
+
+ text can contain special sequences to create colored output. These
+ consist of the escape character \03 and the color name in curly braces,
+ e. g. \03{lightpurple}. \03{default} resets the color.
+
+ Other keyword arguments are passed unchanged to the logger; so far, the
+ only argument that is useful is "exc_info=True", which causes the
+ log message to include an exception traceback.
+
+ """
+ if toStdout: # maintained for backwards-compatibity only
+ logoutput(text, decoder, newline, STDOUT, **kwargs)
+ else:
+ logoutput(text, decoder, newline, INFO, **kwargs)
+
+
+def stdout(text, decoder=None, newline=True, **kwargs):
+ """Output script results to the user via the userinterface."""
+ logoutput(text, decoder, newline, STDOUT, **kwargs)
+
+
+def warning(text, decoder=None, newline=True, **kwargs):
+ """Output a warning message to the user via the userinterface."""
+ logoutput(text, decoder, newline, WARNING, **kwargs)
+
+
+def error(text, decoder=None, newline=True, **kwargs):
+ """Output an error message to the user via the userinterface."""
+ logoutput(text, decoder, newline, ERROR, **kwargs)
+
+
+def log(text, decoder=None, newline=True, **kwargs):
+ """Output a record to the log file."""
+ logoutput(text, decoder, newline, VERBOSE, **kwargs)
+
+
+def critical(text, decoder=None, newline=True, **kwargs):
+ """Output a critical record to the log file."""
+ logoutput(text, decoder, newline, CRITICAL, **kwargs)
+
+
+def debug(text, layer, decoder=None, newline=True, **kwargs):
+ """Output a debug record to the log file.
+
+ @param layer: The name of the logger that text will be sent to.
+ """
+ logoutput(text, decoder, newline, DEBUG, layer, **kwargs)
+
+
+def exception(msg=None, decoder=None, newline=True, tb=False, **kwargs):
+ """Output an error traceback to the user via the userinterface.
+
+ Use directly after an 'except' statement::
+
+ ...
+ except:
+ pywikibot.exception()
+ ...
+
+ or alternatively::
+
+ ...
+ except Exception as e:
+ pywikibot.exception(e)
+ ...
+
+ @param tb: Set to True in order to output traceback also.
+ """
+ if isinstance(msg, BaseException):
+ exc_info = 1
+ else:
+ exc_info = sys.exc_info()
+ msg = u'%s: %s' % (repr(exc_info[1]).split('(')[0],
+ unicode(exc_info[1]).strip())
+ if tb:
+ kwargs['exc_info'] = exc_info
+ logoutput(msg, decoder, newline, ERROR, **kwargs)
diff --git a/pywikibot/page.py b/pywikibot/page.py
index 081c252..2c697c0 100644
--- a/pywikibot/page.py
+++ b/pywikibot/page.py
@@ -16,7 +16,7 @@
#
# Distributed under the terms of the MIT license.
#
-from __future__ import unicode_literals
+from __future__ import absolute_import, unicode_literals
__version__ = '$Id$'
#
diff --git a/pywikibot/tools/__init__.py b/pywikibot/tools/__init__.py
index fa07047..2787b89 100644
--- a/pywikibot/tools/__init__.py
+++ b/pywikibot/tools/__init__.py
@@ -32,15 +32,7 @@
else:
import Queue
-
-def print_debug(msg, *args, **kwargs):
- """Simple debug routine."""
- print(msg)
-
-
-# This variable uses the builtin print function.
-# pywikibot updates it to use logging in bot.init_handlers()
-debug = print_debug
+from pywikibot.logging import debug
_logger = 'tools'
diff --git a/pywikibot/tools/_logging.py b/pywikibot/tools/_logging.py
new file mode 100644
index 0000000..3b07379
--- /dev/null
+++ b/pywikibot/tools/_logging.py
@@ -0,0 +1,135 @@
+# -*- coding: utf-8 -*-
+"""Logging tools."""
+#
+# (C) Pywikibot team, 2009-2015
+#
+# Distributed under the terms of the MIT license.
+#
+from __future__ import absolute_import, unicode_literals
+
+__version__ = '$Id$'
+
+import logging
+import os
+
+from pywikibot.tools import PY2
+
+
+# Logging module configuration
+class RotatingFileHandler(logging.handlers.RotatingFileHandler):
+
+ """Modified RotatingFileHandler supporting unlimited amount of backups."""
+
+ def doRollover(self):
+ """
+ Modified naming system for logging files.
+
+ Overwrites the default Rollover renaming by inserting the count number
+ between file name root and extension. If backupCount is >= 1, the system
+ will successively create new files with the same pathname as the base
+ file, but with inserting ".1", ".2" etc. in front of the filename
+ suffix. For example, with a backupCount of 5 and a base file name of
+ "app.log", you would get "app.log", "app.1.log", "app.2.log", ...
+ through to "app.5.log". The file being written to is always "app.log" -
+ when it gets filled up, it is closed and renamed to "app.1.log", and if
+ files "app.1.log", "app.2.log" etc. already exist, then they are
+ renamed to "app.2.log", "app.3.log" etc. respectively.
+ If backupCount is == -1 do not rotate but create new numbered filenames.
+ The newest file has the highest number except some older numbered files
+ where deleted and the bot was restarted. In this case the ordering
+ starts from the lowest available (unused) number.
+
+ """
+ if self.stream:
+ self.stream.close()
+ self.stream = None
+ root, ext = os.path.splitext(self.baseFilename)
+ if self.backupCount > 0:
+ for i in range(self.backupCount - 1, 0, -1):
+ sfn = "%s.%d%s" % (root, i, ext)
+ dfn = "%s.%d%s" % (root, i + 1, ext)
+ if os.path.exists(sfn):
+ if os.path.exists(dfn):
+ os.remove(dfn)
+ os.rename(sfn, dfn)
+ dfn = "%s.1%s" % (root, ext)
+ if os.path.exists(dfn):
+ os.remove(dfn)
+ os.rename(self.baseFilename, dfn)
+ elif self.backupCount == -1:
+ if not hasattr(self, '_lastNo'):
+ self._lastNo = 1
+ while True:
+ fn = "%s.%d%s" % (root, self._lastNo, ext)
+ self._lastNo += 1
+ if not os.path.exists(fn):
+ break
+ os.rename(self.baseFilename, fn)
+ self.mode = 'w'
+ self.stream = self._open()
+
+ def format(self, record):
+ """Strip trailing newlines before outputting text to file."""
+ # Warnings captured from the warnings system are not processed by
+ # logoutput(), so the 'context' variables are missing.
+ # The same context details are provided by Python 3.X, but need to
+ # be extracted from the warning message for Python <= 2.7.
+ if record.name == 'py.warnings' and 'caller_file' not in record.__dict__:
+ assert len(record.args) == 1, \
+ 'Arguments for record is not correctly set'
+ msg = record.args[0]
+
+ if PY2:
+ record.pathname = msg.partition(':')[0]
+ record.lineno = msg.partition(':')[2].partition(':')[0]
+ record.module = msg.rpartition('/')[2].rpartition('.')[0]
+ else:
+ assert msg.startswith(record.pathname + ':'), \
+ 'Record argument should start with path'
+
+ record.__dict__['caller_file'] = record.pathname
+ record.__dict__['caller_name'] = record.module
+ record.__dict__['caller_line'] = record.lineno
+
+ # Remove the path and the line number, and strip the extra space
+ msg = msg.partition(':')[2].partition(':')[2].lstrip()
+ record.args = (msg,)
+
+ text = logging.handlers.RotatingFileHandler.format(self, record)
+ return text.rstrip("\r\n")
+
+
+class LoggingFormatter(logging.Formatter):
+
+ """Format LogRecords for output to file.
+
+ This formatter *ignores* the 'newline' key of the LogRecord, because
+ every record written to a file must end with a newline, regardless of
+ whether the output to the user's console does.
+
+ """
+
+ def __init__(self, fmt=None, datefmt=None, encoding=None):
+ """Constructor with additional encoding parameter."""
+ logging.Formatter.__init__(self, fmt, datefmt)
+ self._encoding = encoding
+
+ def formatException(self, ei):
+ r"""
+ Convert exception trace to unicode if necessary.
+
+ Make sure that the exception trace is converted to unicode.
+
+ L{exceptions.Error} traces are encoded in our console encoding, which
+ is needed for plainly printing them. However, when logging them
+ using logging.exception, the Python logging module will try to use
+ these traces, and it will fail if they are console encoded strings.
+
+ Formatter.formatException also strips the trailing \n, which we need.
+ """
+ strExc = logging.Formatter.formatException(self, ei)
+
+ if PY2 and isinstance(strExc, bytes):
+ return strExc.decode(self._encoding) + '\n'
+ else:
+ return strExc + '\n'
diff --git a/pywikibot/userinterfaces/gui.py b/pywikibot/userinterfaces/gui.py
index 6ca260b..59faacf 100644
--- a/pywikibot/userinterfaces/gui.py
+++ b/pywikibot/userinterfaces/gui.py
@@ -33,6 +33,8 @@
import pywikibot
+from pywikibot import __url__
+
class TextEditor(ScrolledText):
@@ -517,8 +519,7 @@
except ImportError:
pywikibot.warning('This script requires ImageTk from the'
'Python Imaging Library (PIL).\n'
- 'See: https://www.mediawiki.org/wiki/'
- 'Manual:Pywikibot/flickrripper.py')
+ 'See: {0}/flickrripper.py'.format(__url__))
raise
image = Image.open(photo)
diff --git a/pywikibot/userinterfaces/terminal_interface_base.py b/pywikibot/userinterfaces/terminal_interface_base.py
index 783feb3..3049041 100755
--- a/pywikibot/userinterfaces/terminal_interface_base.py
+++ b/pywikibot/userinterfaces/terminal_interface_base.py
@@ -5,7 +5,7 @@
#
# Distributed under the terms of the MIT license.
#
-from __future__ import unicode_literals
+from __future__ import absolute_import, unicode_literals
__version__ = '$Id$'
diff --git a/tox.ini b/tox.ini
index e4ece8e..6f623b0 100644
--- a/tox.ini
+++ b/tox.ini
@@ -63,6 +63,7 @@
pywikibot/fixes.py \
pywikibot/i18n.py \
pywikibot/logentries.py \
+ pywikibot/logging.py \
pywikibot/login.py \
pywikibot/page.py \
pywikibot/pagegenerators.py \
--
To view, visit https://gerrit.wikimedia.org/r/236502
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I8f1e2c1d09b5d39c5facadb0690ee71bdbf5de1b
Gerrit-PatchSet: 14
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Mattflaschen <mflaschen(a)wikimedia.org>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [FIX] tests: Correctly handle abstract classes
......................................................................
[FIX] tests: Correctly handle abstract classes
The condition for an abstract test class was just that there are no test
classes in the current class it tries to create. So when a test case
subclassed another test case and didn't add more tests it was classified an
abstract class and it skipped a lot of checks, which includes whether it's a
networked test.
Also fix the `MementoTestBase` class as it subclasses just `TestCaseBase` which
doesn't do any magic like automatically determining `net` when for example a
`hostname` is set or subclassing `CheckHostnameMixin` when a `hostname`
attribute is set.
Change-Id: I0bdf4d1ee63e5123b519ba6631a79514bf3d8a7c
---
M tests/aspects.py
M tests/weblinkchecker_tests.py
2 files changed, 28 insertions(+), 15 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/tests/aspects.py b/tests/aspects.py
index 10add8d..570f58f 100644
--- a/tests/aspects.py
+++ b/tests/aspects.py
@@ -657,7 +657,14 @@
for attr_name in dct
if attr_name.startswith('test')]
- dct['abstract_class'] = len(tests) == 0
+ base_tests = []
+ if not tests:
+ for base in bases:
+ base_tests += [attr_name
+ for attr_name, attr in base.__dict__.items()
+ if attr_name.startswith('test') and callable(attr)]
+
+ dct['abstract_class'] = not tests and not base_tests
# Bail out if it is the abstract class.
if dct['abstract_class']:
@@ -716,8 +723,7 @@
if (('sites' not in dct and 'site' not in dct) or
('site' in dct and not dct['site'])):
# Prevent use of pywikibot.Site
- if all(not issubclass(base, DisableSiteMixin) for base in bases):
- bases = tuple([DisableSiteMixin] + list(bases))
+ bases = cls.add_base(bases, DisableSiteMixin)
# 'pwb' tests will _usually_ require a site. To ensure the
# test class dependencies are declarative, this requires the
@@ -751,29 +757,29 @@
# The following section is only processed if the test uses sites.
if 'dry' in dct and dct['dry']:
- bases = tuple([DisconnectedSiteMixin] + list(bases))
+ bases = cls.add_base(bases, DisconnectedSiteMixin)
del dct['net']
else:
dct['net'] = True
if 'cacheinfo' in dct and dct['cacheinfo']:
- bases = tuple([CacheInfoMixin] + list(bases))
+ bases = cls.add_base(bases, CacheInfoMixin)
if 'cached' in dct and dct['cached']:
- bases = tuple([ForceCacheMixin] + list(bases))
+ bases = cls.add_base(bases, ForceCacheMixin)
if 'net' in dct and dct['net']:
- bases = tuple([CheckHostnameMixin] + list(bases))
+ bases = cls.add_base(bases, CheckHostnameMixin)
else:
assert not hostnames, 'net must be True with hostnames defined'
if 'write' in dct and dct['write']:
if 'user' not in dct:
dct['user'] = True
- bases = tuple([SiteWriteMixin] + list(bases))
+ bases = cls.add_base(bases, SiteWriteMixin)
if ('user' in dct and dct['user']) or ('sysop' in dct and dct['sysop']):
- bases = tuple([RequireUserMixin] + list(bases))
+ bases = cls.add_base(bases, RequireUserMixin)
for test in tests:
test_func = dct[test]
@@ -807,6 +813,13 @@
return super(MetaTestCaseClass, cls).__new__(cls, name, bases, dct)
+ @staticmethod
+ def add_base(bases, subclass):
+ """Return a tuple of bases with the subclasses added if not already."""
+ if not any(issubclass(base, subclass) for base in bases):
+ bases = (subclass, ) + bases
+ return bases
+
@add_metaclass
class TestCase(TestTimerMixin, TestCaseBase):
diff --git a/tests/weblinkchecker_tests.py b/tests/weblinkchecker_tests.py
index 7526745..52a1b5a 100644
--- a/tests/weblinkchecker_tests.py
+++ b/tests/weblinkchecker_tests.py
@@ -19,12 +19,12 @@
from scripts import weblinkchecker
-from tests.aspects import unittest, require_modules, TestCase, TestCaseBase
+from tests.aspects import unittest, require_modules, TestCase
from tests import weblib_tests
@require_modules('memento_client')
-class MementoTestBase(TestCaseBase):
+class MementoTestCase(TestCase):
"""Test memento client."""
@@ -39,7 +39,7 @@
self.timegate_uri)
-class WeblibTestMementoInternetArchive(MementoTestBase, weblib_tests.TestInternetArchive):
+class WeblibTestMementoInternetArchive(MementoTestCase, weblib_tests.TestInternetArchive):
"""Test InternetArchive Memento using old weblib tests."""
@@ -47,7 +47,7 @@
hostname = timegate_uri
-class WeblibTestMementoWebCite(MementoTestBase, weblib_tests.TestWebCite):
+class WeblibTestMementoWebCite(MementoTestCase, weblib_tests.TestWebCite):
"""Test WebCite Memento using old weblib tests."""
@@ -55,7 +55,7 @@
hostname = timegate_uri
-class TestMementoWebCite(MementoTestBase):
+class TestMementoWebCite(MementoTestCase):
"""New WebCite Memento tests."""
@@ -70,7 +70,7 @@
self.assertEqual(parsed.netloc, 'www.webcitation.org')
-class TestMementoDefault(MementoTestBase, TestCase):
+class TestMementoDefault(MementoTestCase):
"""Test InternetArchive is default Memento timegate."""
--
To view, visit https://gerrit.wikimedia.org/r/238416
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I0bdf4d1ee63e5123b519ba6631a79514bf3d8a7c
Gerrit-PatchSet: 4
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: interwiki: do not automatically log in
......................................................................
interwiki: do not automatically log in
For interwiki.py, self.site is not necessarily the site we will write
to, and we might want to write to different sites. Interwiki.py uses
the configured usernames to determine which sites we want to write to,
rather than -family: and -code:.
Bug: T108802
Change-Id: Iaedd4ae371ed494c9e448cbe76eb321e7633128f
---
M scripts/interwiki.py
1 file changed, 0 insertions(+), 1 deletion(-)
Approvals:
John Vandenberg: Looks good to me, approved
Malafaya: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/scripts/interwiki.py b/scripts/interwiki.py
index f6b4c5b..e77e471 100755
--- a/scripts/interwiki.py
+++ b/scripts/interwiki.py
@@ -2588,7 +2588,6 @@
hintlessPageGen = pagegenerators.CombinedPageGenerator(
[hintlessPageGen, gen2])
- site.login()
bot = InterwikiBot()
if not hintlessPageGen:
--
To view, visit https://gerrit.wikimedia.org/r/230995
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iaedd4ae371ed494c9e448cbe76eb321e7633128f
Gerrit-PatchSet: 5
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Malafaya <malafaya(a)clix.pt>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [bugfix]: Do not flood log page.
......................................................................
[bugfix]: Do not flood log page.
report_image() has used a regex with re.DOTALL but there where regex strings
with "$" which need re.MULTILINE flag.
Replacing regex by a simple text comparison solves the problem.
- remove all regexes which was used by report_image()
- remove duplicate message part in checkImageOnCommons()
- remove regex parameter in/for report_image()
- create a short_text message from rep_text which cuts attached text in
brackets and strip blanks and line feeds
- check whether page text text_get contains that short_text
Bug: T102467
Change-Id: I3b8f6c7ab65e70b5b48732964a475498083eea44
---
M scripts/checkimages.py
1 file changed, 17 insertions(+), 34 deletions(-)
Approvals:
XZise: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/checkimages.py b/scripts/checkimages.py
index 85cd5a0..b2f29b1 100755
--- a/scripts/checkimages.py
+++ b/scripts/checkimages.py
@@ -916,9 +916,6 @@
return # Image deleted, no hash found. Skip the image.
site = pywikibot.Site('commons', 'commons')
- regexOnCommons = (r"\[\[:File:%s\]\] is also on '''Commons''': "
- r"\[\[commons:File:.*?\]\](?: \(same name\))?$"
- % re.escape(self.imageName))
commons_image_with_this_hash = next(iter(site.allimages(sha1=hash_found,
total=1)), None)
if commons_image_with_this_hash:
@@ -952,21 +949,16 @@
# It's not only on commons but the image needs a check
# the second usually is a url or something like that.
# Compare the two in equal way, both url.
+ repme = (u"\n*[[:File:%s]] is also on '''Commons''': "
+ u"[[commons:File:%s]]"
+ % (self.imageName,
+ commons_image_with_this_hash.title(
+ withNamespace=False)))
if (self.image.title(asUrl=True) ==
commons_image_with_this_hash.title(asUrl=True)):
- repme = (u"\n*[[:File:%s]] is also on '''Commons''': "
- u"[[commons:File:%s]] (same name)"
- % (self.imageName,
- commons_image_with_this_hash.title(
- withNamespace=False)))
- else:
- repme = (u"\n*[[:File:%s]] is also on '''Commons''': "
- u"[[commons:File:%s]]"
- % (self.imageName,
- commons_image_with_this_hash.title(
- withNamespace=False)))
+ repme += " (same name)"
self.report_image(self.imageName, self.rep_page, self.com, repme,
- addings=False, regex=regexOnCommons)
+ addings=False)
return True
def checkImageDuplicated(self, duplicates_rollback):
@@ -979,8 +971,6 @@
duplicates_comment_talk)
dupComment_image = i18n.translate(self.site,
duplicates_comment_image)
- duplicateRegex = (r'\[\[:File:%s\]\] has the following duplicates'
- % re.escape(self.image.title(asUrl=True)))
imagePage = pywikibot.FilePage(self.site, self.imageName)
hash_found = imagePage.latest_file_info.sha1
duplicates = list(self.site.allimages(sha1=hash_found))
@@ -1111,8 +1101,7 @@
repme += u"\n**[[:File:%s]]" % dup_page.title(asUrl=True)
result = self.report_image(self.imageName, self.rep_page,
- self.com, repme, addings=False,
- regex=duplicateRegex)
+ self.com, repme, addings=False)
if not result:
return True # If Errors, exit (but continue the check)
@@ -1123,7 +1112,7 @@
return True # Ok - No problem. Let's continue the checking phase
def report_image(self, image_to_report, rep_page=None, com=None,
- rep_text=None, addings=True, regex=None):
+ rep_text=None, addings=True):
"""Report the files to the report page when needed."""
if not rep_page:
rep_page = self.rep_page
@@ -1135,9 +1124,6 @@
rep_text = self.rep_text
another_page = pywikibot.Page(self.site, rep_page)
-
- if not regex:
- regex = image_to_report
try:
text_get = another_page.get()
except pywikibot.NoPage:
@@ -1159,12 +1145,13 @@
# or not)
return True
- # The talk page includes "_" between the two names, in this way I
- # replace them to " "
- n = re.compile(regex, re.UNICODE | re.DOTALL)
- y = n.findall(text_get)
-
- if y:
+ # Skip if the message is already there.
+ # Don't care for differences inside brackets.
+ end = rep_text.find('(', max(0, rep_text.find(']]')))
+ if end < 0:
+ end = None
+ short_text = rep_text[:end].strip()
+ if short_text in text_get:
pywikibot.output(u"%s is already in the report page."
% image_to_report)
reported = False
@@ -1396,16 +1383,12 @@
% self.imageName + \
"a ''fake license'', license detected: <nowiki>%s</nowiki>" \
% self.license_found
- regexFakeLicense = r"\* ?\[\[:File:%s\]\] seems to have " \
- % (re.escape(self.imageName)) + \
- "a ''fake license'', license detected: <nowiki>%s</nowiki>$" \
- % (re.escape(self.license_found))
printWithTimeZone(
u"%s seems to have a fake license: %s, reporting..."
% (self.imageName, self.license_found))
self.report_image(self.imageName,
rep_text=rep_text_license_fake,
- addings=False, regex=regexFakeLicense)
+ addings=False)
elif self.license_found:
pywikibot.output(u"[[%s]] seems ok, license found: {{%s}}..."
% (self.imageName, self.license_found))
--
To view, visit https://gerrit.wikimedia.org/r/224597
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I3b8f6c7ab65e70b5b48732964a475498083eea44
Gerrit-PatchSet: 7
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Ricordisamoa <ricordisamoa(a)openmailbox.org>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: Upgrade docs dependency to sphinx-epytext-0.0.4
......................................................................
Upgrade docs dependency to sphinx-epytext-0.0.4
sphinx-epytext-0.0.4 fixes a bug allowing '@raise ..:' instead of
'@raises..:'
Also move the doc tox.ini rule above the tool preferences, and
use requests-requirements.txt
Change-Id: Ic084372d573a49deea6ee2dd00b235e08e6eef8d
---
M docs/requirements-py3.txt
M tox.ini
2 files changed, 10 insertions(+), 9 deletions(-)
Approvals:
John Vandenberg: Looks good to me, but someone else must approve
XZise: Looks good to me, approved
jenkins-bot: Verified
diff --git a/docs/requirements-py3.txt b/docs/requirements-py3.txt
index f8b0450..c99a6f2 100644
--- a/docs/requirements-py3.txt
+++ b/docs/requirements-py3.txt
@@ -1,8 +1,7 @@
# This is a PIP requirements file for building Sphinx documentation of pywikibot
# using sphinx-1.3b1 on python3.4, and is used by readthedocs.org.
+# requests-requirements.txt is also needed
sphinx==1.3.1
-sphinx-epytext>=0.0.3
+sphinx-epytext>=0.0.4
-# mandatory; see README.conversion.txt
-requests
diff --git a/tox.ini b/tox.ini
index 1bf0124..e4ece8e 100644
--- a/tox.ini
+++ b/tox.ini
@@ -226,6 +226,14 @@
[testenv:venv]
commands = {posargs}
+[testenv:doc]
+changedir = docs
+commands = make html
+basepython = python3.4
+deps =
+ -rrequests-requirements.txt
+ -rdocs/requirements-py3.txt
+
[flake8]
ignore = E241,E402,E731,FI10,FI11,FI12,FI13,FI15,FI5,P102,P103
exclude = .tox,.git,./*.egg,ez_setup.py,build,externals,user-config.py,./scripts/i18n/*
@@ -237,9 +245,3 @@
ignore = E241,E402,E731
exclude = .tox,.git,./*.egg,ez_setup.py,build,externals,user-config.py,./scripts/i18n/*
max_line_length = 130
-
-[testenv:doc]
-changedir = docs
-commands = make html
-basepython = python3.4
-deps = -rdocs/requirements-py3.txt
--
To view, visit https://gerrit.wikimedia.org/r/238600
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic084372d573a49deea6ee2dd00b235e08e6eef8d
Gerrit-PatchSet: 3
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [FIX] pywikibot.tools.ip: Remove incorrect comment
......................................................................
[FIX] pywikibot.tools.ip: Remove incorrect comment
The comment noted that pywikibot.tools.ip cannot import
`__future__.unicode_literals` since its creation in 25d3fce9, even though
later in 1e54a7d6 this was actually ignored. The comment itself was valid as
that patch broke it, but another solution fixed it a0a194d5 so that the module
itself has no problem in using `unicode_literals` now.
Change-Id: I067218c1fec9555da8cd91d30016523ba863d6f8
(cherry picked from commit 7541f6df68d2bb252ff2b41f7011d6eae9357bb3)
---
M pywikibot/tools/ip.py
1 file changed, 0 insertions(+), 2 deletions(-)
Approvals:
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/tools/ip.py b/pywikibot/tools/ip.py
index 808c0cc..cb4b064 100644
--- a/pywikibot/tools/ip.py
+++ b/pywikibot/tools/ip.py
@@ -5,8 +5,6 @@
#
# Distributed under the terms of the MIT license.
#
-# Note that this module _must_ not import future.unicode_literals
-# otherwise it will not be able to detect the defective ipaddress module.
from __future__ import unicode_literals
__version__ = '$Id$'
--
To view, visit https://gerrit.wikimedia.org/r/238706
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I067218c1fec9555da8cd91d30016523ba863d6f8
Gerrit-PatchSet: 2
Gerrit-Project: pywikibot/core
Gerrit-Branch: 2.0
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [FEAT] site_detect_tests: Inform about duplicates
......................................................................
[FEAT] site_detect_tests: Inform about duplicates
Several URLs have been added to the tests multiple times. And assuming each
test yields the same result running it multiple times won't help. This is
purposefully not cleared after each tests as the same URL may also be in
different tests.
Change-Id: Ifedd148a5662b416fe72db445f719860bd6a95c0
---
M tests/site_detect_tests.py
1 file changed, 18 insertions(+), 0 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/tests/site_detect_tests.py b/tests/site_detect_tests.py
index bcad8cb..4580f41 100644
--- a/tests/site_detect_tests.py
+++ b/tests/site_detect_tests.py
@@ -25,6 +25,11 @@
"""Test Case for MediaWiki detection and site object creation."""
+ # This list is intentionally shared between all classes
+ _urls_tested = set()
+ # Whether it allows multiple tests of the same URL
+ allow_multiple = False
+
def setUp(self):
"""Set up test."""
self.skips = {}
@@ -32,6 +37,8 @@
self.errors = {}
self.passes = {}
self.all = []
+ # reset after end of test
+ self._previous_multiple = self.allow_multiple
super(TestWikiSiteDetection, self).setUp()
def tearDown(self):
@@ -51,6 +58,7 @@
assert 0 <= pos < len(PREFIXES)
return typ, url, res
+ self.allow_multiple = self._previous_multiple
super(TestWikiSiteDetection, self).tearDown()
print('Out of %d sites, %d tests passed, %d tests failed, '
'%d tests skiped and %d tests raised an error'
@@ -70,6 +78,14 @@
def _wiki_detection(self, url, result):
"""Perform one load test."""
self.all += [url]
+ if url in self._urls_tested:
+ msg = 'Testing URL "{0}" multiple times!'.format(url)
+ if self.allow_multiple:
+ print(msg)
+ else:
+ self.errors[url] = msg
+ return
+ self._urls_tested.add(url)
try:
site = MWSite(url)
except (ServerError, Timeout) as e:
@@ -118,6 +134,8 @@
code = 'meta'
net = True
+ allow_multiple = True
+
def test_IWM(self):
"""Test the load_site method for MW sites on the IWM list."""
data = self.get_site().siteinfo['interwikimap']
--
To view, visit https://gerrit.wikimedia.org/r/238701
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifedd148a5662b416fe72db445f719860bd6a95c0
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #2853
Status: Broken
Duration: 28 minutes and 51 seconds
Commit: 0ad7ee6 (master)
Author: Fabian Neundorf
Message: [FIX] doc: Fix syntax issues
Instead of using two single quotes like in wikitext it's using the reST's
syntax to write bold text using asterisks. Also separate lists by one newline
from text.
Change-Id: If2c86ba1ecce69b795646eb56d7104998f0dfeaa
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/9a19dcf854a3...0ad7ee6a…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/80609975
--
You can configure recipients for build notifications in your .travis.yml file. See http://docs.travis-ci.com/user/notifications