Pywikibot-commits January 2015

pywikibot-commits@lists.wikimedia.org

3 participants
206 discussions

[Gerrit] Update License from core - change (pywikibot/compat)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Update License from core ...................................................................... Update License from core Change-Id: Ia1e03ac53f90f3f3f62b35602a2c78b85dae9734 --- M LICENSE 1 file changed, 4 insertions(+), 4 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/LICENSE b/LICENSE index a3fd6e8..724e5a4 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright (c) 2004-2013 Pywikipedia bot team +Copyright (c) 2004-2015 Pywikibot team Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation @@ -35,10 +35,10 @@ about the suitability of this array for any purpose. It is provided "as is" without express or implied warranty. -Everything in the 'externals' directory is general released under seperate -licenses. Some parts may come from Pywikipedia bot team and share the same +Everything in the 'externals' directory is general released under separate +licenses. Some parts may come from Pywikibot team and share the same permissions, but in general they have to be checked for any external package -seperately. +separately. For example the 'BeautifulSoup.py' module is Copyright (c) 2004-2007 Leonard Richardson, and licensed under the terms of the Python Software Foundation license, available at http://www.python.org/download/releases/2.5/license/. -- To view, visit https://gerrit.wikimedia.org/r/186594 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia1e03ac53f90f3f3f62b35602a2c78b85dae9734 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/compat Gerrit-Branch: master Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] flake8: add missing docstrings in category.py - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: flake8: add missing docstrings in category.py ...................................................................... flake8: add missing docstrings in category.py Change-Id: Id1336e8dad468b0a2fb73a979c54489c0710cb4f --- M scripts/category.py 1 file changed, 11 insertions(+), 0 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/category.py b/scripts/category.py index 2b68625..610059d 100755 --- a/scripts/category.py +++ b/scripts/category.py @@ -152,6 +152,7 @@ """ def __init__(self, rebuild=False, filename='category.dump.bz2'): + """Constructor.""" if not os.path.isabs(filename): filename = config.datafilepath(filename) self.filename = filename @@ -182,6 +183,7 @@ self.rebuild() def rebuild(self): + """Rebuild the dabatase.""" self.catContentDB = {} self.superclassDB = {} @@ -220,6 +222,7 @@ return articleset def getSupercats(self, subcat): + """Return the supercategory (or a set of) for a given subcategory.""" self._load() # if we already know which subcategories exist here if subcat in self.superclassDB: @@ -274,6 +277,7 @@ @deprecated_args(editSummary='comment', dry=None) def __init__(self, generator, newcat=None, sort_by_last_name=False, create=False, comment='', follow_redirects=False): + """Constructor.""" super(CategoryAddBot, self).__init__() self.generator = generator self.newcat = newcat @@ -315,6 +319,7 @@ return pywikibot.Page(site, catlink.title()) def treat(self, page): + """Process one page.""" if page.isRedirectPage(): # if it's a redirect use the redirect target instead redirTarget = page.getRedirectTarget() @@ -716,6 +721,7 @@ def __init__(self, catTitle, batchMode=False, editSummary='', useSummaryForDeletion=CategoryMoveRobot.DELETION_COMMENT_AUTOMATIC, titleRegex=None, inPlace=False, pagesonly=False): + """Constructor.""" super(CategoryRemoveRobot, self).__init__( oldcat=catTitle, batch=batchMode, @@ -733,6 +739,7 @@ def __init__(self, catTitle, listTitle, editSummary, overwrite=False, showImages=False, subCats=False, talkPages=False, recurse=False): + """Constructor.""" self.editSummary = editSummary self.overwrite = overwrite self.showImages = showImages @@ -744,6 +751,7 @@ self.recurse = recurse def run(self): + """Start bot.""" setOfArticles = set(self.cat.articles(recurse=self.recurse)) if self.subCats: setOfArticles = setOfArticles.union(set(self.cat.subcategories())) @@ -803,6 +811,7 @@ """ def __init__(self, catTitle, catDB): + """Constructor.""" self.catTitle = catTitle self.catDB = catDB self.site = pywikibot.Site() @@ -925,6 +934,7 @@ flag = True def run(self): + """Start bot.""" cat = pywikibot.Category(self.site, self.catTitle) empty = True @@ -955,6 +965,7 @@ """ def __init__(self, catTitle, catDB, filename=None, maxDepth=10): + """Constructor.""" self.catTitle = catTitle self.catDB = catDB if filename and not os.path.isabs(filename): -- To view, visit https://gerrit.wikimedia.org/r/186585 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id1336e8dad468b0a2fb73a979c54489c0710cb4f Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] Update ui_tests expected values - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Update ui_tests expected values ...................................................................... Update ui_tests expected values - update ansi codes per April 2014 update 1b90718 - update call to input_choice, and use unicode options so it returns unicode Bug: T76248 Change-Id: I8940bdcb764568d1958e1288f32c506f684f08b9 --- M tests/ui_tests.py 1 file changed, 70 insertions(+), 30 deletions(-) Approvals: XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/tests/ui_tests.py b/tests/ui_tests.py index 77d7e62..7045a40 100644 --- a/tests/ui_tests.py +++ b/tests/ui_tests.py @@ -158,6 +158,12 @@ def tearDown(self): unpatch() + def _encode(self, string, encoding='utf-8'): + if sys.version_info[0] > 2: + return string + else: + return string.encode(encoding) + class TestTerminalOutput(UITestCase): """Terminal output tests.""" @@ -267,6 +273,8 @@ """Terminal input tests.""" + input_choice_output = "question ([A]nswer 1, a[n]swer 2, an[s]wer 3) " + def testInput(self): newstdin.write("input to read\n") newstdin.seek(0) @@ -279,53 +287,58 @@ self.assertIsInstance(returned, unicode) self.assertEqual(returned, u"input to read") - @unittest.expectedFailure + def _call_input_choice(self): + rv = pywikibot.input_choice( + "question", + (('answer 1', u'A'), + ('answer 2', u'N'), + ('answer 3', u'S')), + u'A', + automatic_quit=False) + + self.assertEqual(newstdout.getvalue(), "") + + self.assertIsInstance(rv, unicode) + + return rv + def testInputChoiceDefault(self): newstdin.write("\n") newstdin.seek(0) - returned = pywikibot.inputChoice("question", ["answer 1", "answer 2", "answer 3"], ["A", "N", "S"], "A") + returned = self._call_input_choice() - self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "question ([A]nswer 1, a[N]swer 2, an[S]wer 3) ") - - self.assertIsInstance(returned, unicode) self.assertEqual(returned, "a") def testInputChoiceCapital(self): newstdin.write("N\n") newstdin.seek(0) - returned = pywikibot.inputChoice("question", ["answer 1", "answer 2", "answer 3"], ["A", "N", "S"], "A") + returned = self._call_input_choice() - self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "question ([A]nswer 1, a[N]swer 2, an[S]wer 3) ") + self.assertEqual(newstderr.getvalue(), self.input_choice_output) - self.assertIsInstance(returned, unicode) self.assertEqual(returned, "n") def testInputChoiceNonCapital(self): newstdin.write("n\n") newstdin.seek(0) - returned = pywikibot.inputChoice("question", ["answer 1", "answer 2", "answer 3"], ["A", "N", "S"], "A") + returned = self._call_input_choice() - self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "question ([A]nswer 1, a[N]swer 2, an[S]wer 3) ") + self.assertEqual(newstderr.getvalue(), self.input_choice_output) - self.assertIsInstance(returned, unicode) self.assertEqual(returned, "n") def testInputChoiceIncorrectAnswer(self): newstdin.write("X\nN\n") newstdin.seek(0) - returned = pywikibot.inputChoice("question", ["answer 1", "answer 2", "answer 3"], ["A", "N", "S"], "A") + returned = self._call_input_choice() - self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "question ([A]nswer 1, a[N]swer 2, an[S]wer 3) " * 2) + self.assertEqual(newstderr.getvalue(), + self.input_choice_output * 2) - self.assertIsInstance(returned, unicode) self.assertEqual(returned, "n") @unittest.skipUnless(os.name == "posix", "requires Unix console") @@ -333,28 +346,46 @@ """Terminal output color tests.""" + str1 = 'text \03{lightpurple}light purple text\03{default} text' + def testOutputColorizedText(self): - pywikibot.output(u"normal text \03{lightpurple}light purple text\03{default} normal text") + pywikibot.output(self.str1) self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "normal text \x1b[35;1mlight purple text\x1b[0m normal text\n\x1b[0m") + self.assertEqual( + newstderr.getvalue(), + 'text \x1b[95mlight purple text\x1b[0m text\n\x1b[0m') def testOutputNoncolorizedText(self): pywikibot.config.colorized_output = False - pywikibot.output(u"normal text \03{lightpurple}light purple text\03{default} normal text") + pywikibot.output(self.str1) self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "normal text light purple text normal text ***\n") + self.assertEqual( + newstderr.getvalue(), + 'text light purple text text ***\n') + + str2 = ('normal text \03{lightpurple} light purple ' + + '\03{lightblue} light blue \03{default} light purple ' + + '\03{default} normal text') @unittest.expectedFailure def testOutputColorCascade(self): - pywikibot.output(u"normal text \03{lightpurple} light purple \03{lightblue} light blue \03{default} light purple \03{default} normal text") + pywikibot.output(self.str2) self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "normal text \x1b[35;1m light purple \x1b[94;1m light blue \x1b[35;1m light purple \x1b[0m normal text\n\x1b[0m") + self.assertEqual( + newstderr.getvalue(), + 'normal text \x1b[35;1m light purple ' + + '\x1b[94m light blue \x1b[35;1m light purple ' + + '\x1b[0m normal text\n\x1b[0m') def testOutputColorCascade_incorrect(self): """ Test incorrect behavior of testOutputColorCascade. """ - pywikibot.output(u"normal text \03{lightpurple} light purple \03{lightblue} light blue \03{default} light purple \03{default} normal text") + pywikibot.output(self.str2) self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "normal text \x1b[35;1m light purple \x1b[94;1m light blue \x1b[0m light purple \x1b[0m normal text\n\x1b[0m") + self.assertEqual( + newstderr.getvalue(), + 'normal text \x1b[95m light purple ' + + '\x1b[94m light blue \x1b[0m light purple ' + + '\x1b[0m normal text\n\x1b[0m') @unittest.skipUnless(os.name == "posix", "requires Unix console") class TestTerminalUnicodeUnix(UITestCase): @@ -364,16 +395,20 @@ def testOutputUnicodeText(self): pywikibot.output(u"Заглавная_страница") self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), u"Заглавная_страница\n".encode('utf-8')) + self.assertEqual( + newstderr.getvalue(), + self._encode(u'Заглавная_страница\n', 'utf-8')) def testInputUnicodeText(self): - newstdin.write(u"Заглавная_страница\n".encode('utf-8')) + newstdin.write(self._encode(u'Заглавная_страница\n', 'utf-8')) newstdin.seek(0) returned = pywikibot.input(u"Википедию? ") self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), u"Википедию? ".encode('utf-8')) + self.assertEqual( + newstderr.getvalue(), + self._encode(u'Википедию? ', 'utf-8')) self.assertIsInstance(returned, unicode) self.assertEqual(returned, u"Заглавная_страница") @@ -388,7 +423,12 @@ pywikibot.config.transliterate = True pywikibot.output(u"abcd АБГД αβγδ あいうえお") self.assertEqual(newstdout.getvalue(), "") - self.assertEqual(newstderr.getvalue(), "abcd \x1b[33;1mA\x1b[0m\x1b[33;1mB\x1b[0m\x1b[33;1mG\x1b[0m\x1b[33;1mD\x1b[0m \x1b[33;1ma\x1b[0m\x1b[33;1mb\x1b[0m\x1b[33;1mg\x1b[0m\x1b[33;1md\x1b[0m \x1b[33;1ma\x1b[0m\x1b[33;1mi\x1b[0m\x1b[33;1mu\x1b[0m\x1b[33;1me\x1b[0m\x1b[33;1mo\x1b[0m\n\x1b[0m") # noqa + self.assertEqual( + newstderr.getvalue(), + 'abcd \x1b[93mA\x1b[0m\x1b[93mB\x1b[0m\x1b[93mG\x1b[0m' + '\x1b[93mD\x1b[0m \x1b[93ma\x1b[0m\x1b[93mb\x1b[0m\x1b[93mg' + '\x1b[0m\x1b[93md\x1b[0m \x1b[93ma\x1b[0m\x1b[93mi\x1b[0m' + '\x1b[93mu\x1b[0m\x1b[93me\x1b[0m\x1b[93mo\x1b[0m\n\x1b[0m') @unittest.skipUnless(os.name == "nt", "requires Windows console") class WindowsTerminalTestCase(UITestCase): -- To view, visit https://gerrit.wikimedia.org/r/186336 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8940bdcb764568d1958e1288f32c506f684f08b9 Gerrit-PatchSet: 8 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] [FIX] ui tests: Support Python 3 - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [FIX] ui tests: Support Python 3 ...................................................................... [FIX] ui tests: Support Python 3 This uses BytesIO or StringIO depending on the Python version as in Python 3 the sys.std* are supporting strings instead of bytes. The unix output was encoding the text in anyway so this was also fixed. Not all tests succeed because the test conditions haven't been fixed yet. Change-Id: Id79817ac743eb5376c2c8accd028b4e39dfe0e8d --- M pywikibot/userinterfaces/terminal_interface_unix.py M tests/ui_tests.py 2 files changed, 46 insertions(+), 27 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/userinterfaces/terminal_interface_unix.py b/pywikibot/userinterfaces/terminal_interface_unix.py index b980cf6..6b6b282 100755 --- a/pywikibot/userinterfaces/terminal_interface_unix.py +++ b/pywikibot/userinterfaces/terminal_interface_unix.py @@ -47,12 +47,14 @@ # just to be sure, reset the color text += unixColors['default'] - # .encoding does not mean we can write unicode - # to the stream pre-2.7. - if sys.version_info >= (2, 7) and \ - hasattr(targetStream, 'encoding') and \ - targetStream.encoding: - text = text.encode(targetStream.encoding, 'replace').decode(targetStream.encoding) - targetStream.write(text) - else: - targetStream.write(text.encode(self.encoding, 'replace')) + if sys.version_info[0] == 2: + # .encoding does not mean we can write unicode + # to the stream pre-2.7. + if (sys.version_info >= (2, 7) and + hasattr(targetStream, 'encoding') and + targetStream.encoding): + text = text.encode(targetStream.encoding, 'replace').decode( + targetStream.encoding) + else: + text = text.encode(self.encoding, 'replace') + targetStream.write(text) diff --git a/tests/ui_tests.py b/tests/ui_tests.py index f792efc..77d7e62 100644 --- a/tests/ui_tests.py +++ b/tests/ui_tests.py @@ -31,12 +31,28 @@ import os import sys import time -from io import StringIO - -from tests.utils import unittest +import io if sys.version_info[0] > 2: unicode = str + + +class Stream(object): + + """Handler for a StrigIO or BytesIO instance able to patch itself.""" + + def __init__(self, name): + self._stream = io.StringIO() if sys.version_info[0] > 2 else io.BytesIO() + self._name = 'std{0}'.format(name) + self._original = getattr(sys, self._name) + + def patch(self): + setattr(sys, self._name, self._stream) + self._stream.truncate(0) + self._stream.seek(0) + + def unpatch(self): + setattr(sys, self._name, self._original) if os.name == "nt": from multiprocessing.managers import BaseManager @@ -91,31 +107,33 @@ s.serve_forever() if __name__ == "__main__": - oldstderr = sys.stderr - oldstdout = sys.stdout - oldstdin = sys.stdin + strout = Stream('out') + strerr = Stream('err') + strin = Stream('in') - newstdout = StringIO() - newstderr = StringIO() - newstdin = StringIO() + newstdout = strout._stream + newstderr = strerr._stream + newstdin = strin._stream def patch(): """Patch standard terminal files.""" - sys.stdout = newstdout - sys.stderr = newstderr - sys.stdin = newstdin + strout.patch() + strerr.patch() + strin.patch() def unpatch(): """un-patch standard terminal files.""" - sys.stdout = oldstdout - sys.stderr = oldstderr - sys.stdin = oldstdin + strout.unpatch() + strerr.unpatch() + strin.unpatch() try: patch() import pywikibot finally: unpatch() + + from tests.utils import unittest from pywikibot.bot import DEBUG, VERBOSE, INFO, STDOUT, INPUT, WARNING, ERROR, CRITICAL @@ -131,9 +149,6 @@ def setUp(self): patch() - newstdout.truncate(0) - newstderr.truncate(0) - newstdin.truncate(0) pywikibot.config.colorized_output = True pywikibot.config.transliterate = False @@ -525,6 +540,8 @@ unpatch() else: + from tests.utils import unittest + class TestTerminalUI(unittest.TestCase): """Class to show all tests skipped under unittest.""" -- To view, visit https://gerrit.wikimedia.org/r/186542 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id79817ac743eb5376c2c8accd028b4e39dfe0e8d Gerrit-PatchSet: 3 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] Bump copyright notice in LICENSE to 2015 - change (pywikibot/core)

by John Vandenberg (Code Review)

John Vandenberg has submitted this change and it was merged. Change subject: Bump copyright notice in LICENSE to 2015 ...................................................................... Bump copyright notice in LICENSE to 2015 Not doing this for every file, since the devs agreed to only bump the year when substantial changes are made. Change-Id: Icd0e3e335b953a28de7e7697dd7172dcb12bb70b --- M LICENSE 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/LICENSE b/LICENSE index cc87fc4..71d9a57 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,4 @@ -Copyright (c) 2004-2014 Pywikibot team +Copyright (c) 2004-2015 Pywikibot team Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation -- To view, visit https://gerrit.wikimedia.org/r/186352 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Icd0e3e335b953a28de7e7697dd7172dcb12bb70b Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Ricordisamoa <ricordisamoa(a)openmailbox.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] &fixes-help needs to appear after -fixes - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: &fixes-help needs to appear after -fixes ...................................................................... &fixes-help needs to appear after -fixes Change-Id: Id157d6f41e28b2aa1eef3543f2dff8c0953e746a --- M scripts/replace.py 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: John Vandenberg: Looks good to me, approved Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/replace.py b/scripts/replace.py index ee1a67c..95fe144 100755 --- a/scripts/replace.py +++ b/scripts/replace.py @@ -65,12 +65,12 @@ given in the dictionary 'fixes' defined inside the files fixes.py and user-fixes.py. +&fixes-help; + -manualinput Request manual replacements via the command line input even if replacements are already defined. If this option is set (or no replacements are defined via -fix or the arguments) it'll ask for additional replacements at start. - -&fixes-help; -always Don't prompt you for each replacement -- To view, visit https://gerrit.wikimedia.org/r/186333 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id157d6f41e28b2aa1eef3543f2dff8c0953e746a Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 2 months

[Gerrit] [IMPROV] ModuleDeprecationWrapper is module - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [IMPROV] ModuleDeprecationWrapper is module ...................................................................... [IMPROV] ModuleDeprecationWrapper is module It is a replacement for modules so it should be subclass of module. Also sorted the imports alphabetically. Change-Id: If8b2c79f36525bc1c4437bed27c455d8e9f170aa --- M pywikibot/tools.py 1 file changed, 5 insertions(+), 4 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/tools.py b/pywikibot/tools.py index 0eb772e..3d45090 100644 --- a/pywikibot/tools.py +++ b/pywikibot/tools.py @@ -8,12 +8,13 @@ from __future__ import print_function __version__ = '$Id$' +import collections +import inspect +import re import sys import threading import time -import inspect -import re -import collections +import types from distutils.version import Version if sys.version_info[0] > 2: @@ -878,7 +879,7 @@ return call -class ModuleDeprecationWrapper(object): +class ModuleDeprecationWrapper(types.ModuleType): """A wrapper for a module to deprecate classes or variables of it.""" -- To view, visit https://gerrit.wikimedia.org/r/186175 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: If8b2c79f36525bc1c4437bed27c455d8e9f170aa Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 3 months

[Gerrit] [FIX] Http: Allow custom encoding - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [FIX] Http: Allow custom encoding ...................................................................... [FIX] Http: Allow custom encoding This adds the ability to use a specific encodings and warns if the encoding differs. Change-Id: I03609e3d6ec9d8b7f72819358c11d62a792bf4c0 --- M pywikibot/comms/http.py M pywikibot/comms/threadedhttp.py M tests/http_tests.py 3 files changed, 133 insertions(+), 14 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py index 444f3c7..82f5ca3 100644 --- a/pywikibot/comms/http.py +++ b/pywikibot/comms/http.py @@ -222,7 +222,7 @@ @deprecate_arg('ssl', None) -def request(site=None, uri=None, *args, **kwargs): +def request(site=None, uri=None, charset=None, *args, **kwargs): """ Request to Site with default error handling and response decoding. @@ -237,8 +237,12 @@ @type site: L{pywikibot.site.BaseSite} @param uri: the URI to retrieve @type uri: str + @param charset: Either a valid charset (usable for str.decode()) or None + to automatically chose the charset from the returned header (defaults + to latin-1) + @type charset: CodecInfo, str, None @return: The received data - @rtype: unicode + @rtype: a unicode string """ assert(site or uri) if not site: @@ -260,6 +264,7 @@ format_string = kwargs.setdefault("headers", {}).get("user-agent") kwargs["headers"]["user-agent"] = user_agent(site, format_string) + kwargs['charset'] = charset r = fetch(baseuri, *args, **kwargs) return r.content diff --git a/pywikibot/comms/threadedhttp.py b/pywikibot/comms/threadedhttp.py index 8de4213..b7a7422 100644 --- a/pywikibot/comms/threadedhttp.py +++ b/pywikibot/comms/threadedhttp.py @@ -21,6 +21,7 @@ __docformat__ = 'epytext' # standard python libraries +import codecs import re import sys import threading @@ -28,7 +29,6 @@ if sys.version_info[0] > 2: from http import cookiejar as cookielib from urllib.parse import splittype, splithost, unquote, urlparse, urljoin - unicode = str else: import cookielib from urlparse import urlparse, urljoin @@ -337,7 +337,7 @@ """ def __init__(self, uri, method="GET", body=None, headers=None, - callbacks=None, **kwargs): + callbacks=None, charset=None, **kwargs): """ Constructor. @@ -347,6 +347,14 @@ self.method = method self.body = body self.headers = headers + if isinstance(charset, codecs.CodecInfo): + self.charset = charset.name + elif charset: + self.charset = charset + elif headers and 'accept-charset' in headers: + self.charset = headers['accept-charset'] + else: + self.charset = None self.callbacks = callbacks @@ -418,22 +426,60 @@ return self.response_headers.status @property + def header_encoding(self): + """Return charset given by the response header.""" + if not hasattr(self, '_header_encoding'): + pos = self.response_headers['content-type'].find('charset=') + if pos >= 0: + pos += len('charset=') + encoding = self.response_headers['content-type'][pos:] + self._header_encoding = encoding + else: + self._header_encoding = None + return self._header_encoding + + @property def encoding(self): """Detect the response encoding.""" - pos = self.response_headers['content-type'].find('charset=') - if pos >= 0: - pos += len('charset=') - encoding = self.response_headers['content-type'][pos:] - else: - encoding = 'ascii' - # Don't warn, many pages don't contain one - pywikibot.log(u"Http response doesn't contain a charset.") + if not hasattr(self, '_encoding'): + if not self.charset and not self.header_encoding: + pywikibot.log(u"Http response doesn't contain a charset.") + charset = 'latin1' + else: + charset = self.charset + if (self.header_encoding and codecs.lookup(self.header_encoding) != + (codecs.lookup(charset) if charset else None)): + if charset: + pywikibot.warning(u'Encoding "{0}" requested but "{1}" ' + 'received in the header.'.format( + charset, self.header_encoding)) + try: + # TODO: Buffer decoded content, weakref does remove it too + # early (directly after this method) + self.raw.decode(self.header_encoding) + except UnicodeError as e: + self._encoding = e + else: + self._encoding = self.header_encoding + else: + self._encoding = None - return encoding + if charset and (isinstance(self._encoding, Exception) or + not self._encoding): + try: + self.raw.decode(charset) + except UnicodeError as e: + self._encoding = e + else: + self._encoding = charset + + if isinstance(self._encoding, Exception): + raise self._encoding + return self._encoding def decode(self, encoding): """Return the decoded response.""" - return self.raw.decode(encoding) + return self.raw.decode(self.encoding) @property def content(self): diff --git a/tests/http_tests.py b/tests/http_tests.py index 4b1a885..722687f 100644 --- a/tests/http_tests.py +++ b/tests/http_tests.py @@ -310,6 +310,74 @@ self.assertIn('Python/' + str(sys.version_info[0]), http.user_agent()) +class CharsetTestCase(TestCase): + + """Test that HttpRequest correct handles the charsets given.""" + + net = False + + STR = u'äöü' + LATIN1_BYTES = STR.encode('latin1') + UTF8_BYTES = STR.encode('utf8') + + @staticmethod + def _create_request(charset=None, data=UTF8_BYTES): + req = threadedhttp.HttpRequest(None, charset=charset) + req._data = ({'content-type': 'charset=utf-8'}, data[:]) + return req + + def test_no_charset(self): + """Test decoding without explicit charset.""" + req = threadedhttp.HttpRequest(None) + req._data = ({'content-type': ''}, CharsetTestCase.LATIN1_BYTES[:]) + self.assertIsNone(req.charset) + self.assertEqual('latin1', req.encoding) + self.assertEqual(req.raw, CharsetTestCase.LATIN1_BYTES) + self.assertEqual(req.content, CharsetTestCase.STR) + + def test_server_charset(self): + """Test decoding with server explicit charset.""" + req = CharsetTestCase._create_request() + self.assertIsNone(req.charset) + self.assertEqual('utf-8', req.encoding) + self.assertEqual(req.raw, CharsetTestCase.UTF8_BYTES) + self.assertEqual(req.content, CharsetTestCase.STR) + + def test_same_charset(self): + """Test decoding with explicit and equal charsets.""" + req = CharsetTestCase._create_request('utf-8') + self.assertEqual('utf-8', req.charset) + self.assertEqual('utf-8', req.encoding) + self.assertEqual(req.raw, CharsetTestCase.UTF8_BYTES) + self.assertEqual(req.content, CharsetTestCase.STR) + + def test_header_charset(self): + """Test decoding with different charsets and valid header charset.""" + req = CharsetTestCase._create_request('latin1') + self.assertEqual('latin1', req.charset) + self.assertEqual('utf-8', req.encoding) + self.assertEqual(req.raw, CharsetTestCase.UTF8_BYTES) + self.assertEqual(req.content, CharsetTestCase.STR) + + def test_code_charset(self): + """Test decoding with different charsets and invalid header charset.""" + req = CharsetTestCase._create_request('latin1', + CharsetTestCase.LATIN1_BYTES) + self.assertEqual('latin1', req.charset) + self.assertEqual('latin1', req.encoding) + self.assertEqual(req.raw, CharsetTestCase.LATIN1_BYTES) + self.assertEqual(req.content, CharsetTestCase.STR) + + def test_invalid_charset(self): + """Test decoding with different and invalid charsets.""" + req = CharsetTestCase._create_request('utf16', + CharsetTestCase.LATIN1_BYTES) + self.assertEqual('utf16', req.charset) + self.assertRaises(UnicodeDecodeError, lambda: req.encoding) + self.assertEqual(req.raw, CharsetTestCase.LATIN1_BYTES) + self.assertRaises(UnicodeDecodeError, lambda: req.content) + + if __name__ == '__main__': try: unittest.main() -- To view, visit https://gerrit.wikimedia.org/r/158848 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I03609e3d6ec9d8b7f72819358c11d62a792bf4c0 Gerrit-PatchSet: 18 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 3 months

[Gerrit] unicodecsv is a script dependency only on py2 - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: unicodecsv is a script dependency only on py2 ...................................................................... unicodecsv is a script dependency only on py2 Change-Id: I725db1bb06e251e3664d23f88a60acb1ebfb3848 --- M tests/script_tests.py 1 file changed, 4 insertions(+), 1 deletion(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/tests/script_tests.py b/tests/script_tests.py index ce04236..6810a00 100644 --- a/tests/script_tests.py +++ b/tests/script_tests.py @@ -28,15 +28,18 @@ 'script_wui': ['crontab', 'lua'], # Note: package 'lunatic-python' provides module 'lua' - 'data_ingestion': ['unicodecsv'], 'flickrripper': ['flickrapi'], 'match_images': ['PIL.ImageTk'], 'states_redirect': ['pycountry'], } + if sys.version_info < (2, 7): script_deps['replicate_wiki'] = ['argparse'] script_deps['editarticle'] = ['argparse'] +if sys.version_info < (3, 0): + script_deps['data_ingestion'] = ['unicodecsv'] + def check_script_deps(script_name): """Detect whether all dependencies are installed.""" -- To view, visit https://gerrit.wikimedia.org/r/186316 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I725db1bb06e251e3664d23f88a60acb1ebfb3848 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 3 months

[Gerrit] Use setup.py nosetests - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Use setup.py nosetests ...................................................................... Use setup.py nosetests Dependencies are automatically managed by setup.py Change-Id: I5c7d77e77adfdfcaa2258878bce8b49df0f46728 --- M .travis.yml 1 file changed, 3 insertions(+), 3 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/.travis.yml b/.travis.yml index 414475a..fc2a251 100644 --- a/.travis.yml +++ b/.travis.yml @@ -13,7 +13,7 @@ install: - if [[ "$SITE_ONLY" == '1' ]]; then export USE_NOSE=1; fi - - if [[ $TRAVIS_PYTHON_VERSION == '2.6' ]]; then pip install ordereddict unittest2==0.6.0; fi + - if [[ $TRAVIS_PYTHON_VERSION == '2.6' ]]; then pip install unittest2==0.6.0; fi - pip install six - export GITHUB_USER=`echo $TRAVIS_REPO_SLUG | cut -d '/' -f 1` - mkdir ~/.pywikibot @@ -51,9 +51,9 @@ - if [[ "$USE_NOSE" == "1" ]]; then if [[ "$SITE_ONLY" == "1" ]]; then echo "Running site tests only code ${LANGUAGE} on family ${FAMILY}" ; - nosetests tests -v -a "family=$FAMILY,code=$LANGUAGE" ; + python setup.py nosetests --tests tests -v -a "family=$FAMILY,code=$LANGUAGE" ; else - nosetests tests -v ; + python setup.py nosetests --tests tests -v ; fi ; else python setup.py test ; -- To view, visit https://gerrit.wikimedia.org/r/186215 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5c7d77e77adfdfcaa2258878bce8b49df0f46728 Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 3 months

← Newer
1
2
3
4
5
6
7
...
21
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits January 2015