Pywikibot-commits March 2017

pywikibot-commits@lists.wikimedia.org

2 participants
115 discussions

Errored: wikimedia/pywikibot-core#3887 (master - 28e9013)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3887 Status: Errored Duration: 1 hour, 48 minutes, and 57 seconds Commit: 28e9013 (master) Author: jenkins-bot Message: Merge "Localize clean_sandbox.py to Czech" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/bcd2f309cbcd...28e90131… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/207920179 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

7 years, 1 month

Errored: wikimedia/pywikibot-core#3886 (master - bcd2f30)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3886 Status: Errored Duration: 1 hour, 52 minutes, and 37 seconds Commit: bcd2f30 (master) Author: jenkins-bot Message: Merge "Fix ItemClaimFilterPageGenerator" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/ec0f8c24e9df...bcd2f309… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/207897453 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

7 years, 1 month

[Gerrit] pywikibot/core[master]: [IMPR] Provide a new generator which yields a subclass of Page

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/316321 ) Change subject: [IMPR] Provide a new generator which yields a subclass of Page ...................................................................... [IMPR] Provide a new generator which yields a subclass of Page - PageClassGenerator yields objects of Page, FilePage, Category or User which depends on the namespace found for the page. - redirect the FileGenerator and CategoryGenerator to this new generator and deprecate them by using redirect_func. - Now you haven't care about the page class coming from generator as before. You always get the right type. It is up to the developer to use them accordingly. Change-Id: Ib9107277474c603b2e687f5573afa57f2071af99 --- M pywikibot/pagegenerators.py 1 file changed, 19 insertions(+), 20 deletions(-) Approvals: Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index e60c98f..95a9384 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -1,4 +1,4 @@ -# -*- coding: utf-8 -*- +# -*- coding: utf-8 -*- """ This module offers a wide variety of page generators. @@ -14,7 +14,7 @@ &params; """ # -# (C) Pywikibot team, 2008-2016 +# (C) Pywikibot team, 2008-2017 # # Distributed under the terms of the MIT license. # @@ -1784,26 +1784,22 @@ return itertools.chain(*generators) -def CategoryGenerator(generator): - """Yield pages from another generator as Category objects. +def PageClassGenerator(generator): + """ + Yield pages from another generator as Page subclass objects. - Makes sense only if it is ascertained that only categories are being - retrieved. - + The page class type depends on the page namespace. + Objects may be Category, FilePage, Userpage or Page. """ for page in generator: - yield pywikibot.Category(page) - - -def FileGenerator(generator): - """ - Yield pages from another generator as FilePage objects. - - Makes sense only if it is ascertained - that only images are being retrieved. - """ - for page in generator: - yield pywikibot.FilePage(page) + if page.namespace() == page.site.namespaces.USER: + yield pywikibot.User(page) + elif page.namespace() == page.site.namespaces.FILE: + yield pywikibot.FilePage(page) + elif page.namespace() == page.site.namespaces.CATEGORY: + yield pywikibot.Category(page) + else: + yield page def PageWithTalkPageGenerator(generator, return_talk_only=False): @@ -2780,7 +2776,10 @@ # Deprecated old names available for compatibility with compat. -ImageGenerator = redirect_func(FileGenerator, old_name='ImageGenerator') +ImageGenerator = redirect_func(PageClassGenerator, old_name='ImageGenerator') +FileGenerator = redirect_func(PageClassGenerator, old_name='FileGenerator') +CategoryGenerator = redirect_func(PageClassGenerator, + old_name='CategoryGenerator') UnCategorizedTemplatesGenerator = redirect_func( UnCategorizedTemplateGenerator, old_name='UnCategorizedTemplatesGenerator') RecentchangesPageGenerator = redirect_func( -- To view, visit https://gerrit.wikimedia.org/r/316321 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ib9107277474c603b2e687f5573afa57f2071af99 Gerrit-PatchSet: 7 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years, 1 month

Errored: wikimedia/pywikibot-core#3885 (master - ec0f8c2)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3885 Status: Errored Duration: 51 minutes and 43 seconds Commit: ec0f8c2 (master) Author: jenkins-bot Message: Merge "Improve and re-enable translateMagicWords" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/45c57f7a9303...ec0f8c24… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/207896871 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

7 years, 1 month

[Gerrit] pywikibot/core[master]: Localize clean_sandbox.py to Czech

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/338379 ) Change subject: Localize clean_sandbox.py to Czech ...................................................................... Localize clean_sandbox.py to Czech Change-Id: I44dd1fba26f29c65aaf5076b79997bbe29f21721 --- M scripts/clean_sandbox.py 1 file changed, 4 insertions(+), 1 deletion(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/scripts/clean_sandbox.py b/scripts/clean_sandbox.py index 5b6b409..fcbb427 100755 --- a/scripts/clean_sandbox.py +++ b/scripts/clean_sandbox.py @@ -58,7 +58,10 @@ 'az': u'\n{{Qaralama dəftəri}}\n' u'', 'bar': '{{Bitte erst NACH dieser Zeile schreiben! (Begrüßungskasten)}}\n', - 'cs': u'{{subst:/uhrabat}}', + 'cs': '{{Tento řádek neměňte}}\n\n\n' + "== Bábovičky ==\n#'''první'''\n#''druhá''\n*třetí\n" + "*'''''čtvrtá'''''\n pátá\n;šestá\n:sedmá", 'da': u'{{subst:Sandkasse tekst}}', 'de': u'{{subst:Wikipedia:Spielwiese/Vorlage}}', 'en': u'{{Sandbox heading}}\n<!-- Hello! Feel free to try your formatting ' -- To view, visit https://gerrit.wikimedia.org/r/338379 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I44dd1fba26f29c65aaf5076b79997bbe29f21721 Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years, 1 month

Errored: wikimedia/pywikibot-core#3884 (master - 45c57f7)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3884 Status: Errored Duration: 1 hour, 57 minutes, and 48 seconds Commit: 45c57f7 (master) Author: jenkins-bot Message: Merge "djvu.py: added features and refactored" View the changeset: https://github.com/wikimedia/pywikibot-core/compare/d3780953aa37...45c57f7a… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/207880684 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

7 years, 1 month

Errored: wikimedia/pywikibot-core#3883 (master - d378095)

by Travis CI

Build Update for wikimedia/pywikibot-core ------------------------------------- Build: #3883 Status: Errored Duration: 2 hours, 3 minutes, and 13 seconds Commit: d378095 (master) Author: dalba Message: test_CategoryFilterPageGenerator: Increase the expected length by two Two new pages have recently been added to the category. Bug: T159628 Change-Id: I7c9f1f117aa8813e643f5af9c3847e9badb9d77b View the changeset: https://github.com/wikimedia/pywikibot-core/compare/785fee8bab6b...d3780953… View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/207878498 -- You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications

7 years, 1 month

[Gerrit] pywikibot/core[master]: Fix ItemClaimFilterPageGenerator

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/341143 ) Change subject: Fix ItemClaimFilterPageGenerator ...................................................................... Fix ItemClaimFilterPageGenerator If negate argument is provided, the if clause is always false and kills the generator. Change-Id: I2de6036ab3fc4295990472531b5b68f618ce7644 --- M pywikibot/pagegenerators.py M tests/pagegenerators_tests.py 2 files changed, 11 insertions(+), 4 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index 29ddced..06fa7de 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -1537,7 +1537,7 @@ @type negate: bool """ for page in generator: - if cls.__filter_match(page, prop, claim, qualifiers) and not negate: + if cls.__filter_match(page, prop, claim, qualifiers) is not negate: yield page diff --git a/tests/pagegenerators_tests.py b/tests/pagegenerators_tests.py index 133caf8..dbcad04 100755 --- a/tests/pagegenerators_tests.py +++ b/tests/pagegenerators_tests.py @@ -666,7 +666,7 @@ """Test item claim filter page generator generator.""" - def _simple_claim_test(self, prop, claim, qualifiers, valid): + def _simple_claim_test(self, prop, claim, qualifiers, valid, negate=False): """ Test given claim on sample (India) page. @@ -678,10 +678,12 @@ @param valid: true if the page should be yielded by the generator, false otherwise @type valid: bool + @param negate: true to swap the filters' behavior + @type negate: bool """ item = pywikibot.ItemPage(self.get_repo(), 'Q668') - gen = pagegenerators.ItemClaimFilterPageGenerator([item], prop, - claim, qualifiers) + gen = pagegenerators.ItemClaimFilterPageGenerator([item], prop, claim, + qualifiers, negate) pages = set(gen) self.assertEqual(len(pages), 1 if valid else 0) @@ -732,6 +734,11 @@ self._simple_claim_test('P625', '21,77', None, True) self._simple_claim_test('P625', '21,78.05,0.01', None, False) + def test_negative_filter(self): + """Test negative ItemClaimFilterPageGenerator.""" + self._simple_claim_test('P463', 'Q37470', None, False, True) + self._simple_claim_test('P463', 'Q37471', None, True, True) + class TestFactoryGenerator(DefaultSiteTestCase): -- To view, visit https://gerrit.wikimedia.org/r/341143 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2de6036ab3fc4295990472531b5b68f618ce7644 Gerrit-PatchSet: 3 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years, 1 month

[Gerrit] pywikibot/core[master]: Improve and re-enable translateMagicWords

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/322401 ) Change subject: Improve and re-enable translateMagicWords ...................................................................... Improve and re-enable translateMagicWords Bug: T151107 Change-Id: I97b1bc227959feb889610c3c2b7c9609b0c5dccf --- M pywikibot/cosmetic_changes.py M tests/cosmetic_changes_tests.py 2 files changed, 22 insertions(+), 8 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/cosmetic_changes.py b/pywikibot/cosmetic_changes.py index f1923a8..07fb95c 100755 --- a/pywikibot/cosmetic_changes.py +++ b/pywikibot/cosmetic_changes.py @@ -50,7 +50,7 @@ """ # # (C) xqt, 2009-2016 -# (C) Pywikibot team, 2006-2016 +# (C) Pywikibot team, 2006-2017 # # Distributed under the terms of the MIT license. # @@ -221,8 +221,7 @@ self.cleanUpSectionHeaders, self.putSpacesInLists, self.translateAndCapitalizeNamespaces, - # FIXME: fix bugs and re-enable - # self.translateMagicWords, + self.translateMagicWords, self.replaceDeprecatedTemplates, # FIXME: fix bugs and re-enable # self.resolveHtmlEntities, @@ -406,19 +405,22 @@ """Use localized magic words.""" # not wanted at ru # arz uses english stylish codes - if self.site.code not in ['arz', 'ru']: + # no need to run on English wikis + if self.site.code not in ['arz', 'en', 'ru']: exceptions = ['nowiki', 'comment', 'math', 'pre'] for magicWord in ['img_thumbnail', 'img_left', 'img_center', 'img_right', 'img_none', 'img_framed', 'img_frameless', 'img_border', 'img_upright', ]: - aliases = self.site.getmagicwords(magicWord) + aliases = list(self.site.getmagicwords(magicWord)) + preferred = aliases.pop(0) if not aliases: continue text = textlib.replaceExcept( text, - r'\[\[(?P<left>.+?:.+?\..+?\|) *(' + '|'.join(aliases) + - r') *(?P<right>(\|.*?)?\]\])', - r'[[\g<left>' + aliases[0] + r'\g<right>', exceptions) + r'\[\[(?P<left>.+?:.+?\..+?\|(.*?\|)*?) *(' + + '|'.join(aliases) + + r') *(?P<right>(\|.*?)*?\]\])', + r'[[\g<left>' + preferred + r'\g<right>', exceptions) return text def cleanUpLinks(self, text): diff --git a/tests/cosmetic_changes_tests.py b/tests/cosmetic_changes_tests.py index cc3947c..5db7281 100644 --- a/tests/cosmetic_changes_tests.py +++ b/tests/cosmetic_changes_tests.py @@ -244,6 +244,18 @@ self.assertEqual( '[[File:Foo.bar|miniatur]]', self.cct.translateMagicWords('[[File:Foo.bar|mini]]')) + self.assertEqual( + '[[File:Foo.bar|links|miniatur]]', + self.cct.translateMagicWords('[[File:Foo.bar|left|thumb]]')) + self.assertEqual( + '[[File:Foo.bar|250px|links]]', + self.cct.translateMagicWords('[[File:Foo.bar|250px|left]]')) + self.assertEqual( + '[[File:Foo.bar|links|250px]]', + self.cct.translateMagicWords('[[File:Foo.bar| left |250px]]')) + self.assertEqual( + '[[File:Foo.bar|250px|zentriert|Baz]]', + self.cct.translateMagicWords('[[File:Foo.bar|250px|center|Baz]]')) def test_cleanUpLinks_pipes(self): """Test cleanUpLinks method.""" -- To view, visit https://gerrit.wikimedia.org/r/322401 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I97b1bc227959feb889610c3c2b7c9609b0c5dccf Gerrit-PatchSet: 5 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years, 1 month

[Gerrit] pywikibot/core[master]: djvu.py: added features and refactored

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/339906 ) Change subject: djvu.py: added features and refactored ...................................................................... djvu.py: added features and refactored Added: - cache control decorator - page number check decorator - retrieval of page info from djvu file Refactored: - tiny wrapper of subprocess Popen() to reduce code repetition Added tests. Renamed test file myfile.djvu to myfilé.djvu to test non-ascii filenames. Change-Id: Idf465abe0f9aab3d7c213098ae02335269740ecf --- M pywikibot/tools/djvu.py R tests/data/djvu/myfilé.djvu M tests/djvu_tests.py 3 files changed, 219 insertions(+), 45 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/tools/djvu.py b/pywikibot/tools/djvu.py index da7c595..1eee4e3 100644 --- a/pywikibot/tools/djvu.py +++ b/pywikibot/tools/djvu.py @@ -2,7 +2,7 @@ # -*- coding: utf-8 -*- """Wrapper around djvulibre to access djvu files properties and content.""" # -# (C) Pywikibot team, 2015-2016 +# (C) Pywikibot team, 2015-2017 # # Distributed under the terms of the MIT license. # @@ -10,10 +10,54 @@ __version__ = '$Id$' -import os.path +import collections +import os +import re import subprocess -from pywikibot.logging import error +import pywikibot + +from pywikibot.tools import ( + deprecated, deprecated_args, + StringTypes, +) + + +def _call_cmd(args, lib='djvulibre'): + """ + Tiny wrapper around subprocess.Popen(). + + @param args: same as Popen() + @type args: sequence or string + + @param library: library to be logged in logging messages + @type library: string + + @param log: log process output; errors are always logged. + @type library: bool + + + @return: returns a tuple (res, stdoutdata), where + res is True if dp.returncode != 0 else False + """ + if not isinstance(args, StringTypes): + # upcast if any param in sequence args is not in StringTypes + args = [str(a) if not isinstance(a, StringTypes) else a for a in args] + cmd = ' '.join(args) + else: + cmd = args + + dp = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdoutdata, stderrdata = dp.communicate() + + if dp.returncode != 0: + pywikibot.error('{0} error; {1}'.format(lib, cmd)) + pywikibot.error('{0}'.format(stderrdata)) + return (False, stdoutdata) + + pywikibot.log('SUCCESS: {0} (PID: {1})'.format(cmd, dp.pid)) + + return (True, stdoutdata) class DjVuFile(object): @@ -28,39 +72,143 @@ """ - def __init__(self, file_djvu): + @deprecated_args(file_djvu='file') + def __init__(self, file): """ Constructor. - @param file_djvu: filename (including path) to djvu file - @type file_djvu: string/unicode + @param file: filename (including path) to djvu file + @type file: string/unicode """ - file_djvu = os.path.expanduser(file_djvu) + file = os.path.expanduser(file) + file = os.path.abspath(file) # Check file exists and has read permissions. - with open(file_djvu): - self.file_djvu = file_djvu + with open(file): + self.file = file + self.dirname = os.path.dirname(file) - def number_of_images(self): - """Return the (cached) number of images in the djvu file.""" - if not hasattr(self, '_image_count'): - dp = subprocess.Popen(['djvused', '-e', 'n', self.file_djvu], - stdout=subprocess.PIPE, stderr=subprocess.PIPE) - (stdoutdata, stderrdata) = dp.communicate() - if dp.returncode != 0: - error('djvulibre library error!\n%s' % stderrdata) - self._image_count = int(stdoutdata) - return self._image_count + # pattern for parsing of djvudump output. + self._pat_form = re.compile( + r' *?FORM:DJVU *?\[\d+\] *?(?P<id>{[^\}]*?})? *?\[P(?P<n>\d+)\]') + self._pat_info = re.compile(r'DjVu.*?(?P<size>\d+x\d+).*?(?P<dpi>\d+) dpi') - def has_text(self): - """Test if the djvu file has a text-layer.""" + @property + @deprecated('DjVuFile.file') + def file_djvu(self): + """Deprecated file_djvu instance variable.""" + return self.file + + def check_cache(fn): # flake8: disable=N805 + """Decorator to check if cache shall be cleared.""" + cache = ['_page_count', '_has_text', '_page_info'] + + def wrapper(obj, *args, **kwargs): + force = kwargs.get('force', False) + if force: + for el in cache: + obj.__dict__.pop(el, None) + _res = fn(obj, *args, **kwargs) + return _res + return wrapper + + def check_page_number(fn): # flake8: disable=N805 + """Decorator to check if page number is valid. + + @raises ValueError + """ + def wrapper(obj, *args, **kwargs): + n = args[0] + force = kwargs.get('force', False) + if not (1 <= n <= obj.number_of_images(force=force)): + raise ValueError('Page %d not in file %s [%d-%d]' + % (n, obj.file, n, obj.number_of_images())) + _res = fn(obj, *args, **kwargs) + return _res + return wrapper + + @check_cache + def number_of_images(self, force=False): + """ + Return the number of images in the djvu file. + + @param force: if True, refresh the cached data + @type force: bool + """ + if not hasattr(self, '_page_count'): + res, stdoutdata = _call_cmd(['djvused', '-e', 'n', self.file]) + if not res: + return False + self._page_count = int(stdoutdata) + return self._page_count + + @check_page_number + def page_info(self, n, force=False): + """ + Return a tuple (id, (size, dpi)) for page n of djvu file. + + @param force: if True, refresh the cached data + @type force: bool + """ + if not hasattr(self, '_page_info'): + self._get_page_info(force=force) + return self._page_info[n] + + @check_cache + def _get_page_info(self, force=False): + """ + Return a dict of tuples (id, (size, dpi)) for all pages of djvu file. + + @param force: if True, refresh the cached data + @type force: bool + """ + if not hasattr(self, '_page_info'): + self._page_info = {} + + res, stdoutdata = _call_cmd(['djvudump', self.file]) + if not res: + return False + + has_text = False + for line in stdoutdata.decode('utf-8').split('\n'): + if 'TXTz' in line: + has_text = True + + if 'FORM:DJVU' in line: + m = self._pat_form.search(line) + if m: + key, id = int(m.group('n')), m.group('id') + else: + key, id = '', 1 + + if 'INFO' in line: + m = self._pat_info.search(line) + if m: + size, dpi = m.group('size'), int(m.group('dpi')) + else: + size, dpi = None, None + else: + continue + + self._page_info[key] = (id, (size, dpi)) + self._has_text = has_text + return self._page_info + + def get_most_common_info(self): + """Return most common size and dpi for pages in djvu file.""" + cnt = collections.Counter(s_d for _, s_d in self._get_page_info().values()) + (size, dpi), _ = cnt.most_common()[0] + return size, dpi + + @check_cache + def has_text(self, force=False): + """ + Test if the djvu file has a text-layer. + + @param force: if True, refresh the cached data + @type force: bool + """ if not hasattr(self, '_has_text'): - dp = subprocess.Popen(['djvudump', self.file_djvu], - stdout=subprocess.PIPE, stderr=subprocess.PIPE) - (stdoutdata, stderrdata) = dp.communicate() - if dp.returncode != 0: - error('djvulibre library error!\n%s' % stderrdata) - txt = stdoutdata.decode('utf-8') - self._has_text = 'TXTz' in txt + self._get_page_info(force=force) return self._has_text def _remove_control_chars(self, data): @@ -79,17 +227,23 @@ txt = txt.strip('\x0c\n ') return txt - def get_page(self, n): - """Get page n for djvu file.""" - if not self.has_text(): - raise ValueError('Djvu file %s has no text layer.' % self.file_djvu) - if not (1 <= n <= self.number_of_images()): - raise ValueError('Requested page number %d is not in file %s' - ' page range [%d-%d]' - % (n, self.file_djvu, 1, self.number_of_images())) - dp = subprocess.Popen(['djvutxt', '--page=%d' % n, self.file_djvu], - stdout=subprocess.PIPE, stderr=subprocess.PIPE) - (stdoutdata, stderrdata) = dp.communicate() - if dp.returncode != 0: - error('djvulibre library error!\n%s' % stderrdata) + @check_page_number + @check_cache + def get_page(self, n, force=False): + """ + Get page n for djvu file. + + @param force: if True, refresh the cached data + @type force: bool + """ + if not self.has_text(force=force): + raise ValueError('Djvu file %s has no text layer.' % self.file) + res, stdoutdata = _call_cmd(['djvutxt', '--page=%d' % n, self.file]) + if not res: + return False return self._remove_control_chars(stdoutdata) + + # This is to be used only if this class is subclassed and the decorators + # needs to be used by the child. + check_page_number = staticmethod(check_page_number) + check_cache = staticmethod(check_cache) diff --git a/tests/data/djvu/myfile.djvu "b/tests/data/djvu/myfil\303\251.djvu" old mode 100755 new mode 100644 similarity index 100% rename from tests/data/djvu/myfile.djvu rename to "tests/data/djvu/myfil\303\251.djvu" Binary files differ diff --git a/tests/djvu_tests.py b/tests/djvu_tests.py index ce38022..31a6fb0 100644 --- a/tests/djvu_tests.py +++ b/tests/djvu_tests.py @@ -1,9 +1,9 @@ #!/usr/bin/python # -*- coding: utf-8 -*- -"""Unit tests for djvutext.py script.""" +"""Unit tests for djvu.py.""" # -# (C) Pywikibot team, 2015 +# (C) Pywikibot team, 2017 # # Distributed under the terms of the MIT license. # @@ -27,7 +27,7 @@ net = False file_djvu_not_existing = join_djvu_data_path('not_existing.djvu') - file_djvu = join_djvu_data_path('myfile.djvu') + file_djvu = join_djvu_data_path('myfilé.djvu') # test non-ascii name file_djvu_wo_text = join_djvu_data_path('myfile_wo_text.djvu') test_txt = 'A file with non-ASCII characters, \nlike é or ç' @@ -44,13 +44,24 @@ def test_file_existance(self): """Test file existence checks.""" djvu = DjVuFile(self.file_djvu) - self.assertEqual(self.file_djvu, djvu.file_djvu) + self.assertEqual(self.file_djvu, djvu.file) self.assertRaises(IOError, DjVuFile, self.file_djvu_not_existing) def test_number_of_images(self): """Test page number generator.""" djvu = DjVuFile(self.file_djvu) self.assertEqual(djvu.number_of_images(), 4) + + def test_page_info(self): + """Test page info retrieval.""" + djvu = DjVuFile(self.file_djvu) + self.assertEqual(djvu.page_info(1), + ('{myfile.djvu}', ('1092x221', 600))) + + def test_get_most_common_info(self): + """Test page number generator.""" + djvu = DjVuFile(self.file_djvu) + self.assertEqual(djvu.get_most_common_info(), ('1092x221', 600)) def test_has_text(self): """Test if djvu file contains text.""" @@ -78,6 +89,15 @@ self.assertFalse(djvu.has_text()) self.assertRaises(ValueError, djvu.get_page, 100) + def test_clear_cache(self): + """Test if djvu file contains text.""" + djvu = DjVuFile(self.file_djvu) + self.assertTrue(djvu.has_text()) + djvu._has_text = False + self.assertFalse(djvu.has_text()) + self.assertTrue(djvu.has_text(force=True)) + + if __name__ == '__main__': # pragma: no cover try: unittest.main() -- To view, visit https://gerrit.wikimedia.org/r/339906 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Idf465abe0f9aab3d7c213098ae02335269740ecf Gerrit-PatchSet: 5 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years, 1 month

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits March 2017