jenkins-bot has submitted this change and it was merged.
Change subject: Check a valid WebCite memento link
......................................................................
Check a valid WebCite memento link
The WebCite timegate URL returns a 400 error if a URI is not provided.
Use a valid timemap URL instead, as it has a small response payload.
Bug: T128998
Change-Id: I9cb473279bb95d056751a3ce41fe809fb62ed283
---
M tests/weblinkchecker_tests.py
1 file changed, 4 insertions(+), 2 deletions(-)
Approvals:
John Vandenberg: Looks good to me, but someone else must approve
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/tests/weblinkchecker_tests.py b/tests/weblinkchecker_tests.py
index 2080d98..5e052e2 100644
--- a/tests/weblinkchecker_tests.py
+++ b/tests/weblinkchecker_tests.py
@@ -52,7 +52,8 @@
"""Test WebCite Memento using old weblib tests."""
timegate_uri = 'http://timetravel.mementoweb.org/webcite/timegate/'
- hostname = timegate_uri
+ hostname = ('http://timetravel.mementoweb.org/webcite/'
+ 'timemap/json/http://google.com')
class TestMementoWebCite(MementoTestCase):
@@ -60,7 +61,8 @@
"""New WebCite Memento tests."""
timegate_uri = 'http://timetravel.mementoweb.org/webcite/timegate/'
- hostname = timegate_uri
+ hostname = ('http://timetravel.mementoweb.org/webcite/'
+ 'timemap/json/http://google.com')
def test_newest(self):
"""Test WebCite for newest https://google.com."""
--
To view, visit https://gerrit.wikimedia.org/r/277213
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I9cb473279bb95d056751a3ce41fe809fb62ed283
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #3329
Status: Errored
Duration: 55 minutes and 50 seconds
Commit: d02a9bd (master)
Author: AbdealiJK
Message: pywikibot.tools: Raise error if bz2 not found
bz2 is required in pywikibot to open compressed files. But,
sometimes python is not compiled with bz2. This causes pywikibot
to fail while it is being imported.
This commit shows a warning in such cases and tries to use
bz2file as an alternative. If both are not found, it raises an
error lazily (i.e. only when bz2 is needed to open a compressed
file).
Bug: T123092
Change-Id: Iace28a5b356cff19118983fd089770c75a1b4dff
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/e229be8fe221...d02a9bd6…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/116041688
--
You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications
jenkins-bot has submitted this change and it was merged.
Change subject: pywikibot.tools: Raise error if bz2 not found
......................................................................
pywikibot.tools: Raise error if bz2 not found
bz2 is required in pywikibot to open compressed files. But,
sometimes python is not compiled with bz2. This causes pywikibot
to fail while it is being imported.
This commit shows a warning in such cases and tries to use
bz2file as an alternative. If both are not found, it raises an
error lazily (i.e. only when bz2 is needed to open a compressed
file).
Bug: T123092
Change-Id: Iace28a5b356cff19118983fd089770c75a1b4dff
---
M pywikibot/tools/__init__.py
M requirements.txt
M setup.py
M tests/tools_tests.py
4 files changed, 49 insertions(+), 3 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/tools/__init__.py b/pywikibot/tools/__init__.py
index c2c5bba..0c5bcee 100644
--- a/pywikibot/tools/__init__.py
+++ b/pywikibot/tools/__init__.py
@@ -8,7 +8,6 @@
from __future__ import absolute_import, print_function, unicode_literals
__version__ = '$Id$'
-import bz2
import collections
import gzip
import inspect
@@ -38,6 +37,15 @@
from pywikibot.logging import debug
+try:
+ import bz2
+except ImportError as bz2_import_error:
+ try:
+ import bz2file as bz2
+ warn('package bz2 was not found; using bz2file', ImportWarning)
+ except ImportError:
+ warn('package bz2 and bz2file were not found', ImportWarning)
+ bz2 = bz2_import_error
if PYTHON_VERSION < (3, 5):
# although deprecated in 3 completely no message was emitted until 3.5
@@ -988,6 +996,8 @@
extension = ''
if extension == 'bz2':
+ if isinstance(bz2, ImportError):
+ raise bz2
return wrap(bz2.BZ2File(filename, mode), 1)
elif extension == 'gz':
return wrap(gzip.open(filename, mode), 0)
diff --git a/requirements.txt b/requirements.txt
index 1e6945e..3658030 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -91,3 +91,8 @@
# scripts/weblinkchecker.py
memento_client>=0.5.1
+
+# tools/__init__.py
+# pywikibot prefers using the inbuilt bz2 module if python was compiled with
+# bz2 support. But if it wasn't, bz2file is used instead.
+# bz2file
diff --git a/setup.py b/setup.py
index f56eab6..97f121a 100644
--- a/setup.py
+++ b/setup.py
@@ -41,7 +41,7 @@
if not python_is_supported():
raise RuntimeError(versions_required_message % sys.version)
-test_deps = []
+test_deps = ['bz2file']
dependencies = ['requests']
@@ -141,6 +141,12 @@
script_deps['data_ingestion.py'] = extra_deps['csv']
+try:
+ import bz2 # noqa: unused import
+except ImportError:
+ # Use bz2file if the python is not compiled with bz2 support.
+ dependencies.append('bz2file')
+
# Some of the ui_tests depend on accessing the console window's menu
# to set the console font and copy and paste, achieved using pywinauto
# which depends on pywin32.
diff --git a/tests/tools_tests.py b/tests/tools_tests.py
index 2d0935e..ed039b5 100644
--- a/tests/tools_tests.py
+++ b/tests/tools_tests.py
@@ -20,7 +20,9 @@
from pywikibot import tools
from tests import join_xml_data_path
-from tests.aspects import unittest, DeprecationTestCase, TestCase, MetaTestCaseClass
+from tests.aspects import (
+ unittest, require_modules, DeprecationTestCase, TestCase, MetaTestCaseClass
+)
from tests.utils import expected_failure_if, add_metaclass
@@ -106,6 +108,29 @@
self.assertEqual(self._get_content(self.base_file + '.bz2', use_extension=False),
self.original_content)
+ @require_modules('bz2file')
+ def test_open_archive_with_bz2file(self):
+ """Test open_archive when bz2file library."""
+ old_bz2 = tools.bz2
+ try:
+ tools.bz2 = __import__('bz2file')
+ self.assertEqual(self._get_content(self.base_file + '.bz2'),
+ self.original_content)
+ self.assertEqual(self._get_content(self.base_file + '.bz2',
+ use_extension=False),
+ self.original_content)
+ finally:
+ tools.bz2 = old_bz2
+
+ def test_open_archive_without_bz2(self):
+ """Test open_archive when bz2 and bz2file are not available."""
+ old_bz2 = tools.bz2
+ try:
+ tools.bz2 = ImportError()
+ self.assertRaises(ImportError, self._get_content, self.base_file + '.bz2')
+ finally:
+ tools.bz2 = old_bz2
+
def test_open_archive_gz(self):
"""Test open_archive with gz compressor in the standard library."""
self.assertEqual(self._get_content(self.base_file + '.gz'), self.original_content)
--
To view, visit https://gerrit.wikimedia.org/r/277207
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Iace28a5b356cff19118983fd089770c75a1b4dff
Gerrit-PatchSet: 9
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: AbdealiJK <abdealikothari(a)gmail.com>
Gerrit-Reviewer: AbdealiJK <abdealikothari(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de>
Gerrit-Reviewer: jenkins-bot <>
Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #3328
Status: Errored
Duration: 1 hour, 21 minutes, and 14 seconds
Commit: e229be8 (master)
Author: darthbhyrava
Message: Fix: test_no_qualifiers failing due to Coordinate's precision
Return None instead of a ValueError when both
dim and precision are set to None in Coordinate.
Bug: T129834
Change-Id: I258629a6e75207322a94d5312e7eb6225f11c572
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/999c171b8463...e229be8f…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/116030474
--
You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications
jenkins-bot has submitted this change and it was merged.
Change subject: Fix: test_no_qualifiers failing due to Coordinate's precision
......................................................................
Fix: test_no_qualifiers failing due to Coordinate's precision
Return None instead of a ValueError when both
dim and precision are set to None in Coordinate.
Bug: T129834
Change-Id: I258629a6e75207322a94d5312e7eb6225f11c572
---
M pywikibot/__init__.py
M tests/wikibase_tests.py
2 files changed, 1 insertion(+), 3 deletions(-)
Approvals:
John Vandenberg: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py
index 74ad87c..c80c9e1 100644
--- a/pywikibot/__init__.py
+++ b/pywikibot/__init__.py
@@ -355,7 +355,7 @@
@rtype: float or None
"""
if self._dim is None and self._precision is None:
- raise ValueError('No values set for dim or precision')
+ return None
if self._precision is None and self._dim is not None:
radius = 6378137 # TODO: Support other globes
self._precision = math.degrees(
diff --git a/tests/wikibase_tests.py b/tests/wikibase_tests.py
index a0d8265..7b747e3 100644
--- a/tests/wikibase_tests.py
+++ b/tests/wikibase_tests.py
@@ -155,8 +155,6 @@
self.assertIsInstance(y.precision, float)
z = pywikibot.Coordinate(site=repo, lat=12.0, lon=13.0)
with self.assertRaises(ValueError):
- z.precision
- with self.assertRaises(ValueError):
z.precisionToDim()
def test_WbTime(self):
--
To view, visit https://gerrit.wikimedia.org/r/277277
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I258629a6e75207322a94d5312e7eb6225f11c572
Gerrit-PatchSet: 6
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Darthbhyrava <hbhyrava(a)gmail.com>
Gerrit-Reviewer: Darthbhyrava <hbhyrava(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged.
Change subject: [bugfix] Re-enable stopme() called inside a script
......................................................................
[bugfix] Re-enable stopme() called inside a script
- re-implement stopme() and _flush() methods from compat.
- stopme() may be called inside some scripts to drop the process
from the throttle log after pending threads finish. It does not
clear async_manager.
- _flush() is called at exit time. It finishes pending threads,
clears the async_manager by sending a None request to the queue
and drops the process from throttle log.
- global variable 'stopped' is omitted
- show debug message with 'flush()' instead of 'stopme()'
- remaining() counts the remaining pages to be put.
Use its result for colorized message.
- leave the _flush loop when queue is empty. Otherwise the the script hangs.
Bug: T85061
Change-Id: I62db390d503a780ac8193cc499aca13111ea97f0
---
M pywikibot/__init__.py
1 file changed, 40 insertions(+), 28 deletions(-)
Approvals:
Mpaa: Looks good to me, approved
Xqt: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py
index f9e750c..74ad87c 100644
--- a/pywikibot/__init__.py
+++ b/pywikibot/__init__.py
@@ -1,7 +1,7 @@
# -*- coding: utf-8 -*-
"""The initialization file for the Pywikibot framework."""
#
-# (C) Pywikibot team, 2008-2015
+# (C) Pywikibot team, 2008-2016
#
# Distributed under the terms of the MIT license.
#
@@ -861,47 +861,59 @@
# Throttle and thread handling
-stopped = False
-
def stopme():
"""
Drop this process from the throttle log, after pending threads finish.
- Can be called manually if desired, but if not, will be called automatically
- at Python exit.
+ Can be called manually if desired. Does not clean async_manager.
+ This should be run when a bot does not interact with the Wiki, or
+ when it has stopped doing so. After a bot has run stopme() it will
+ not slow down other bots any more.
"""
- global stopped
+ _flush(False)
+
+
+def _flush(stop=True):
+ """
+ Drop this process from the throttle log, after pending threads finish.
+
+ Wait for the page-putter to flush its queue. Also drop this process from the
+ throttle log. Called automatically at Python exit.
+ """
_logger = "wiki"
- if not stopped:
- debug(u"stopme() called", _logger)
+ debug('_flush() called', _logger)
- def remaining():
- remainingPages = page_put_queue.qsize() - 1
+ def remaining():
+ remainingPages = page_put_queue.qsize()
+ if stop:
# -1 because we added a None element to stop the queue
+ remainingPages -= 1
- remainingSeconds = datetime.timedelta(
- seconds=(remainingPages * config.put_throttle))
- return (remainingPages, remainingSeconds)
+ remainingSeconds = datetime.timedelta(
+ seconds=(remainingPages * config.put_throttle))
+ return (remainingPages, remainingSeconds)
+ if stop:
+ # None task element leaves async_manager
page_put_queue.put((None, [], {}))
- stopped = True
- if page_put_queue.qsize() > 1:
- num, sec = remaining()
- output(color_format(
- '{lightblue}Waiting for {num} pages to be put. '
- 'Estimated time remaining: {sec}{default}', num=num, sec=sec))
+ num, sec = remaining()
+ if num > 0:
+ output(color_format(
+ '{lightblue}Waiting for {num} pages to be put. '
+ 'Estimated time remaining: {sec}{default}', num=num, sec=sec))
- while(_putthread.isAlive()):
- try:
- _putthread.join(1)
- except KeyboardInterrupt:
- if input_yn('There are %i pages remaining in the queue. '
- 'Estimated time remaining: %s\nReally exit?'
- % remaining(), default=False, automatic_quit=False):
- return
+ while _putthread.isAlive() and page_put_queue.qsize() > 0:
+ try:
+ _putthread.join(1)
+ except KeyboardInterrupt:
+ if input_yn('There are {0} pages remaining in the queue. '
+ 'Estimated time remaining: {1}\nReally exit?'
+ ''.format(*remaining()),
+ default=False, automatic_quit=False):
+ return
# only need one drop() call because all throttles use the same global pid
try:
@@ -910,7 +922,7 @@
except IndexError:
pass
-atexit.register(stopme)
+atexit.register(_flush)
# Create a separate thread for asynchronous page saves (and other requests)
--
To view, visit https://gerrit.wikimedia.org/r/276428
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I62db390d503a780ac8193cc499aca13111ea97f0
Gerrit-PatchSet: 4
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>