Pywikibot-commits June 2018

pywikibot-commits@lists.wikimedia.org

1 participants
71 discussions

[Gerrit] pywikibot/core[master]: Remove an unused attribute from HarvestRobot

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/440547 ) Change subject: Remove an unused attribute from HarvestRobot ...................................................................... Remove an unused attribute from HarvestRobot The attribute is only used to make a list of templates. Change-Id: Id247b3359b51f493a59738e1d5f01523c11786e8 --- M scripts/harvest_template.py M tox.ini 2 files changed, 6 insertions(+), 6 deletions(-) Approvals: Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/harvest_template.py b/scripts/harvest_template.py index 2db474e..5bb82bc 100755 --- a/scripts/harvest_template.py +++ b/scripts/harvest_template.py @@ -123,14 +123,14 @@ """A bot to add Wikidata claims.""" - def __init__(self, generator, templateTitle, fields, **kwargs): + def __init__(self, generator, template_title, fields, **kwargs): """ Initializer. @param generator: A generator that yields Page objects @type generator: iterator - @param templateTitle: The template to work on - @type templateTitle: str + @param template_title: The template to work on + @type template_title: str @param fields: A dictionary of fields that are of use to us @type fields: dict @keyword islink: Whether non-linked values should be treated as links @@ -149,7 +149,6 @@ }) super(HarvestRobot, self).__init__(**kwargs) self.generator = generator - self.templateTitle = templateTitle.replace(u'_', u' ') # TODO: Make it a list which also includes the redirects to the template self.fields = {} for key, value in fields.items(): @@ -158,7 +157,8 @@ else: # backwards compatibility self.fields[key] = (value, PropertyOptionHandler()) self.cacheSources() - self.templateTitles = self.getTemplateSynonyms(self.templateTitle) + template_title = template_title.replace('_', ' ') + self.templateTitles = self.getTemplateSynonyms(template_title) self.linkR = textlib.compileLinkR() self.create_missing_item = self.getOption('create') diff --git a/tox.ini b/tox.ini index bef79b1..df5d598 100644 --- a/tox.ini +++ b/tox.ini @@ -209,7 +209,7 @@ scripts/data_ingestion.py : N803, N806 scripts/fixing_redirects.py : N803, N806 scripts/flickrripper.py : T001, N803, N806 - scripts/harvest_template.py : T001, N803 + scripts/harvest_template.py : T001 scripts/imagecopy.py : N801, N803, N806 scripts/imagecopy_self.py : N801, N803, N806 scripts/imagerecat.py : N803, N806 -- To view, visit https://gerrit.wikimedia.org/r/440547 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id247b3359b51f493a59738e1d5f01523c11786e8 Gerrit-Change-Number: 440547 Gerrit-PatchSet: 3 Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

[Gerrit] pywikibot/core[master]: [IMPR] use tools.has_module for checking installed modules

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/433374 ) Change subject: [IMPR] use tools.has_module for checking installed modules ...................................................................... [IMPR] use tools.has_module for checking installed modules Change-Id: I45388a056ae2e2c79950a6b34f14c71e11663ce0 --- M tests/script_tests.py 1 file changed, 5 insertions(+), 9 deletions(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/tests/script_tests.py b/tests/script_tests.py index cf539c1..22d1593 100644 --- a/tests/script_tests.py +++ b/tests/script_tests.py @@ -10,10 +10,7 @@ import os import sys -from pywikibot.tools import ( - PY2, - StringTypes, -) +from pywikibot.tools import has_module, PY2, StringTypes from tests import join_root_path, unittest_print from tests.aspects import (unittest, DefaultSiteTestCase, MetaTestCaseClass, @@ -49,11 +46,10 @@ """Detect whether all dependencies are installed.""" if script_name in script_deps: for package_name in script_deps[script_name]: - try: - __import__(package_name) - except ImportError as e: - unittest_print('%s depends on %s, which isnt available:\n%s' - % (script_name, package_name, e)) + if not has_module(package_name): + unittest_print( + "{} depends on {}, which isn't available" + .format(script_name, package_name)) return False return True -- To view, visit https://gerrit.wikimedia.org/r/433374 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I45388a056ae2e2c79950a6b34f14c71e11663ce0 Gerrit-Change-Number: 433374 Gerrit-PatchSet: 5 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

[Gerrit] pywikibot/core[master]: Fix claimit to work on Wikidata as well

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/440291 ) Change subject: Fix claimit to work on Wikidata as well ...................................................................... Fix claimit to work on Wikidata as well The generator might yield pages from Wikidata, then the page is None and this generates AttributeError. Change-Id: I7b3f8535aaae433f8016a62e2288aaf8d8a74961 --- M scripts/claimit.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/claimit.py b/scripts/claimit.py index 300076d..3c523f8 100755 --- a/scripts/claimit.py +++ b/scripts/claimit.py @@ -94,8 +94,9 @@ """Treat each page.""" for claim in self.claims: # The generator might yield pages from multiple sites + site = page.site if page is not None else None self.user_add_claim_unless_exists( - item, claim, self.exists_arg, page.site) + item, claim, self.exists_arg, site) def main(*args): -- To view, visit https://gerrit.wikimedia.org/r/440291 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7b3f8535aaae433f8016a62e2288aaf8d8a74961 Gerrit-Change-Number: 440291 Gerrit-PatchSet: 1 Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

[Gerrit] pywikibot/core[master]: [fix] Don't show wrong empty generator message in category.py

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/439553 ) Change subject: [fix] Don't show wrong empty generator message in category.py ...................................................................... [fix] Don't show wrong empty generator message in category.py Bug: T196871 Change-Id: I94be5d408691c0ce35a59a27081b5ec2e5ae78d6 --- M pywikibot/bot.py M scripts/category.py 2 files changed, 4 insertions(+), 1 deletion(-) Approvals: Dvorapa: Looks good to me, but someone else must approve Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/bot.py b/pywikibot/bot.py index 1557b9d..0a65ca4 100644 --- a/pywikibot/bot.py +++ b/pywikibot/bot.py @@ -1210,6 +1210,7 @@ self._treat_counter = 0 self._save_counter = 0 + self._generator_completed = False @property def current_page(self): @@ -1509,6 +1510,8 @@ '\n%s: page count reached Python 2 sys.maxint (%d).\n' 'Python 3 should be used to process very large batches' % (self.__class__.__name__, sys.maxint)) + else: + self._generator_completed = True except QuitKeyboardInterrupt: pywikibot.output('\nUser quit %s bot run...' % self.__class__.__name__) diff --git a/scripts/category.py b/scripts/category.py index efaeee6..35d53dd 100755 --- a/scripts/category.py +++ b/scripts/category.py @@ -1215,7 +1215,7 @@ def teardown(self): """Cleanups after run operation.""" - if not self._treat_counter: + if self._generator_completed and not self._treat_counter: pywikibot.output('There are no pages or files in category {}' .format(self.cat_title)) -- To view, visit https://gerrit.wikimedia.org/r/439553 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I94be5d408691c0ce35a59a27081b5ec2e5ae78d6 Gerrit-Change-Number: 439553 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

[Gerrit] pywikibot/core[master]: dev-requirements.txt: Update pytest requirement

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/440264 ) Change subject: dev-requirements.txt: Update pytest requirement ...................................................................... dev-requirements.txt: Update pytest requirement The new release of pytest-timeout requires pytest>=3.6.0. Bug: T197202 Change-Id: I4ef6c2c16e36e24d5b57c10b09fd61bfbe757539 --- M dev-requirements.txt 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/dev-requirements.txt b/dev-requirements.txt index 01bca1c..750585a 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -2,7 +2,7 @@ # unittest2==0.8.0 ; python_full_version == '2.7.2' -pytest>=2.8.0 +pytest>=3.6.0 pytest-timeout pytest-runner pytest-cov -- To view, visit https://gerrit.wikimedia.org/r/440264 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4ef6c2c16e36e24d5b57c10b09fd61bfbe757539 Gerrit-Change-Number: 440264 Gerrit-PatchSet: 1 Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

[Gerrit] pywikibot/core[master]: Split GeneratorFactory.handleArg into several handler methods

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/440096 ) Change subject: Split GeneratorFactory.handleArg into several handler methods ...................................................................... Split GeneratorFactory.handleArg into several handler methods handleArg was implemented using a long chain elif blocks. Such blocks have negative impact on performance and code complexity; they also make debugging harder. With this new implementation, the old format of logevent arugments (-<logevent>log) is completely ignored when the log type is unknown the handleArg returnes False. Previously it used raise NotImplementedError. This change should not be a problem since all the other unknown arguments are also treated this way. Update test_logevents_parse accordingly. Change-Id: I844c781d9bc80c30734d2f82def24db8633b7f0d --- M pywikibot/pagegenerators.py M tests/pagegenerators_tests.py 2 files changed, 510 insertions(+), 379 deletions(-) Approvals: Dalba: Looks good to me, approved Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index 108c434..ab81b8d 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -675,6 +675,483 @@ return LogeventsPageGenerator(logtype, user or None, total=total, start=start, end=end) + def _handle_filelinks(self, value): + """Handle `-filelinks` argument.""" + if not value: + value = i18n.input( + 'pywikibot-enter-file-links-processing', + fallback_prompt='Links to which file page should be ' + 'processed?') + if not value.startswith(self.site.namespace(6) + ':'): + value = 'Image:' + value + page = pywikibot.FilePage(self.site, value) + return FileLinksGenerator(page) + + def _handle_linter(self, value): + """Handle `-linter` argument.""" + if not self.site.has_extension('Linter'): + raise UnknownExtension( + '-linter needs a site with Linter extension.') + cats = self.site.siteinfo.get('linter') # Get linter categories. + valid_cats = [c for _list in cats.values() for c in _list] + + value = '' if value is None else value + cat, sep, lint_from = value.partition('/') + if not lint_from: + lint_from = None + + if cat == 'show': # Display categories of lint errors. + _i = ' ' * 4 + txt = 'Available categories of lint errors:\n' + for prio, _list in cats.items(): + txt += '{indent}{prio}\n'.format(indent=_i, prio=prio) + for c in _list: + txt += '{indent}{cat}\n'.format(indent=2 * _i, cat=c) + pywikibot.output('%s' % txt) + return True + + if not cat: + lint_cats = valid_cats + elif cat in ['low', 'medium', 'high']: + lint_cats = cats[cat] + else: + lint_cats = cat.split(',') + for lint_cat in lint_cats: + if lint_cat not in valid_cats: + raise ValueError('Invalid category of lint errors: %s' + % cat) + + return self.site.linter_pages( + lint_categories='|'.join(lint_cats), namespaces=self.namespaces, + lint_from=lint_from) + + def _handle_unusedfiles(self, value): + """Handle `-unusedfiles` argument.""" + return UnusedFilesGenerator(total=_int_none(value), site=self.site) + + def _handle_lonelypages(self, value): + """Handle `-lonelypages` argument.""" + return LonelyPagesPageGenerator(total=_int_none(value), site=self.site) + + def _handle_unwatched(self, value): + """Handle `-unwatched` argument.""" + return UnwatchedPagesPageGenerator( + total=_int_none(value), site=self.site) + + def _handle_property(self, value): + """Handle `-property` argument.""" + if not value: + question = 'Which property name to be used?' + value = pywikibot.input(question + ' (List [?])') + pnames = self.site.get_property_names() + # also use the default by <enter> key + if value in '?' or value not in pnames: + for i, item in enumerate(pnames, start=1): + pywikibot.output( + '{0:{1}}: {2}'.format(i, len(str(len(pnames))), + item)) + prefix, value = pywikibot.input_choice( + question, ListOption(self.site.get_property_names())) + return page_with_property_generator(value, site=self.site) + + def _handle_usercontribs(self, value): + """Handle `-usercontribs` argument.""" + return UserContributionsGenerator(value, site=self.site) + + def _handle_withoutinterwiki(self, value): + """Handle `-withoutinterwiki` argument.""" + return WithoutInterwikiPageGenerator( + total=_int_none(value), site=self.site) + + def _handle_interwiki(self, value): + """Handle `-interwiki` argument.""" + if not value: + value = i18n.input( + 'pywikibot-enter-page-processing', + fallback_prompt='Which page should be processed?') + page = pywikibot.Page(pywikibot.Link(value, self.site)) + return InterwikiPageGenerator(page) + + def _handle_randomredirect(self, value): + """Handle `-randomredirect` argument.""" + # partial workaround for bug T119940 + # to use -namespace/ns with -randomredirect, -ns must be given + # before -randomredirect + # otherwise default namespace is 0 + namespaces = self.namespaces or 0 + return RandomRedirectPageGenerator( + total=_int_none(value), site=self.site, namespaces=namespaces) + + def _handle_random(self, value): + """Handle `-random` argument.""" + # partial workaround for bug T119940 + # to use -namespace/ns with -random, -ns must be given + # before -random + # otherwise default namespace is 0 + namespaces = self.namespaces or 0 + return RandomPageGenerator( + total=_int_none(value), site=self.site, namespaces=namespaces) + + def _handle_recentchanges(self, value): + """Handle `-recentchanges` argument.""" + rcstart = None + rcend = None + rctag = None + total = None + params = value.split(',') if value else [] + if params and not params[0].isdigit(): + rctag = params.pop(0) + if len(params) == 2: + offset = float(params[0]) + duration = float(params[1]) + if offset < 0 or duration < 0: + raise ValueError('Negative valued parameters passed.') + elif len(params) > 2: + raise ValueError('More than two parameters passed.') + else: + total = int(params[0]) if params else 60 + if len(params) == 2: + ts_time = self.site.server_time() + rcstart = ts_time + timedelta(minutes=-(offset + duration)) + rcend = ts_time + timedelta(minutes=-offset) + return RecentChangesPageGenerator( + namespaces=self.namespaces, total=total, start=rcstart, end=rcend, + site=self.site, reverse=True, tag=rctag, + _filter_unique=self._filter_unique) + + def _handle_liverecentchanges(self, value): + """Handle `-liverecentchanges` argument.""" + self.nopreload = True + return LiveRCPageGenerator(site=self.site, total=_int_none(value)) + + def _handle_file(self, value): + """Handle `-file` argument.""" + if not value: + value = pywikibot.input('Please enter the local file name:') + return TextfilePageGenerator(value, site=self.site) + + def _handle_namespaces(self, value): + """Handle `-namespaces` argument.""" + if isinstance(self._namespaces, frozenset): + warn('Cannot handle arg -namespaces as namespaces can not ' + 'be altered after a generator is created.', + ArgumentDeprecationWarning, 2) + return True + if not value: + value = pywikibot.input('What namespace are you filtering on?') + NOT_KEY = 'not:' + if value.startswith(NOT_KEY): + value = value[len(NOT_KEY):] + resolve = self.site.namespaces.resolve + not_ns = set(resolve(value.split(','))) + if not self._namespaces: + self._namespaces = list( + set(self.site.namespaces.values()) - not_ns) + else: + self._namespaces = list( + set(resolve(self._namespaces)) - not_ns) + else: + self._namespaces += value.split(',') + return True + + _handle_ns = _handle_namespaces + _handle_namespace = _handle_namespaces + + def _handle_limit(self, value): + """Handle `-limit` argument.""" + if not value: + value = pywikibot.input('What is the limit value?') + self.limit = _int_none(value) + return True + + def _handle_category(self, value): + """Handle `-category` argument.""" + return self.getCategoryGen( + value, recurse=False, gen_func=CategorizedPageGenerator) + + _handle_cat = _handle_category + + def _handle_catr(self, value): + """Handle `-catr` argument.""" + return self.getCategoryGen( + value, recurse=True, gen_func=CategorizedPageGenerator) + + def _handle_subcats(self, value): + """Handle `-subcats` argument.""" + return self.getCategoryGen( + value, recurse=False, gen_func=SubCategoriesPageGenerator) + + def _handle_subcatsr(self, value): + """Handle `-subcatsr` argument.""" + return self.getCategoryGen( + value, recurse=True, gen_func=SubCategoriesPageGenerator) + + def _handle_catfilter(self, value): + """Handle `-catfilter` argument.""" + cat, _ = self.getCategory(value) + self.catfilter_list.append(cat) + return True + + def _handle_page(self, value): + """Handle `-page` argument.""" + if not value: + value = pywikibot.input('What page do you want to use?') + return [pywikibot.Page(pywikibot.Link(value, self.site))] + + def _handle_pageid(self, value): + """Handle `-pageid` argument.""" + if not value: + value = pywikibot.input('What pageid do you want to use?') + return PagesFromPageidGenerator(value, site=self.site) + + def _handle_uncatfiles(self, value): + """Handle `-uncatfiles` argument.""" + return UnCategorizedImageGenerator(site=self.site) + + def _handle_uncatcat(self, value): + """Handle `-uncatcat` argument.""" + return UnCategorizedCategoryGenerator(site=self.site) + + def _handle_uncat(self, value): + """Handle `-uncat` argument.""" + return UnCategorizedPageGenerator(site=self.site) + + def _handle_ref(self, value): + """Handle `-ref` argument.""" + if not value: + value = pywikibot.input( + 'Links to which page should be processed?') + page = pywikibot.Page(pywikibot.Link(value, self.site)) + return ReferringPageGenerator(page) + + def _handle_links(self, value): + """Handle `-links` argument.""" + if not value: + value = pywikibot.input( + 'Links from which page should be processed?') + page = pywikibot.Page(pywikibot.Link(value, self.site)) + return LinkedPageGenerator(page) + + def _handle_weblink(self, value): + """Handle `-weblink` argument.""" + if not value: + value = pywikibot.input( + 'Pages with which weblink should be processed?') + # If url is * we make it None in order to search for every page + # with any URL. + if value == '*': + value = None + return LinksearchPageGenerator(value, site=self.site) + + def _handle_transcludes(self, value): + """Handle `-transcludes` argument.""" + if not value: + value = pywikibot.input( + 'Pages that transclude which page should be processed?') + page = pywikibot.Page(pywikibot.Link(value, + defaultNamespace=10, + source=self.site)) + return ReferringPageGenerator(page, onlyTemplateInclusion=True) + + def _handle_start(self, value): + """Handle `-start` argument.""" + if not value: + value = '!' + firstpagelink = pywikibot.Link(value, self.site) + return self.site.allpages( + start=firstpagelink.title, namespace=firstpagelink.namespace, + filterredir=False) + + def _handle_prefixindex(self, value): + """Handle `-prefixindex` argument.""" + if not value: + value = pywikibot.input('What page names are you looking for?') + return PrefixingPageGenerator(prefix=value, site=self.site) + + def _handle_newimages(self, value): + """Handle `-newimages` argument.""" + return NewimagesPageGenerator(total=_int_none(value), site=self.site) + + def _handle_newpages(self, value): + """Handle `-newpages` argument.""" + # partial workaround for bug T69249 + # to use -namespace/ns with -newpages, -ns must be given + # before -newpages + # otherwise default namespace is 0 + namespaces = self.namespaces or 0 + return NewpagesPageGenerator( + namespaces=namespaces, total=_int_none(value), site=self.site) + + def _handle_unconnectedpages(self, value): + """Handle `-unconnectedpages` argument.""" + return self.site.unconnected_pages(total=_int_none(value)) + + def _handle_imagesused(self, value): + """Handle `-imagesused` argument.""" + if not value: + value = pywikibot.input( + 'Images on which page should be processed?') + page = pywikibot.Page(pywikibot.Link(value, self.site)) + return ImagesPageGenerator(page) + + def _handle_searchitem(self, value): + """Handle `-searchitem` argument.""" + if not value: + value = pywikibot.input('Text to look for:') + params = value.split(':') + value = params[-1] + lang = params[0] if len(params) == 2 else None + return WikibaseSearchItemPageGenerator( + value, language=lang, site=self.site) + + def _handle_search(self, value): + """Handle `-search` argument.""" + if not value: + value = pywikibot.input('What do you want to search for?') + # In order to be useful, all namespaces are required + return SearchPageGenerator(value, namespaces=[], site=self.site) + + @staticmethod + def _handle_google(value): + """Handle `-google` argument.""" + return GoogleSearchPageGenerator(value) + + def _handle_titleregex(self, value): + """Handle `-titleregex` argument.""" + if not value: + value = pywikibot.input( + 'What page names are you looking for?') + self.titlefilter_list.append(value) + return True + + def _handle_titleregexnot(self, value): + """Handle `-titleregexnot` argument.""" + if not value: + value = pywikibot.input( + 'All pages except which ones?') + self.titlenotfilter_list.append(value) + return True + + def _handle_grep(self, value): + """Handle `-grep` argument.""" + if not value: + value = pywikibot.input('Which pattern do you want to grep?') + self.articlefilter_list.append(value) + return True + + def _handle_ql(self, value): + """Handle `-ql` argument.""" + if not self.site.has_extension('ProofreadPage'): + raise UnknownExtension( + 'Ql filtering needs a site with ProofreadPage extension.') + value = [int(_) for _ in value.split(',')] + if min(value) < 0 or max(value) > 4: # Invalid input ql. + valid_ql = [ + '{0}: {1}'.format(*i) + for i in self.site.proofread_levels.items()] + valid_ql = ', '.join(valid_ql) + pywikibot.warning('Acceptable values for -ql are:\n %s' + % valid_ql) + self.qualityfilter_list = value + return True + + def _handle_onlyif(self, value): + """Handle `-onlyif` argument.""" + return self._onlyif_onlyifnot_handler(value, False) + + def _handle_onlyifnot(self, value): + """Handle `-onlyifnot` argument.""" + return self._onlyif_onlyifnot_handler(value, True) + + def _onlyif_onlyifnot_handler(self, value, ifnot): + """Handle `-onlyif` and `-onlyifnot` arguments.""" + if not value: + value = pywikibot.input('Which claim do you want to filter?') + p = re.compile(r'(?<!\\),') # Match "," only if there no "\" before + temp = [] # Array to store split argument + for arg in p.split(value): + temp.append(arg.replace(r'\,', ',').split('=')) + self.claimfilter_list.append( + (temp[0][0], temp[0][1], dict(temp[1:]), ifnot)) + return True + + def _handle_yahoo(self, value): + """Handle `-yahoo` argument.""" + return YahooSearchPageGenerator(value, site=self.site) + + @staticmethod + def _handle_untagged(value): + """Handle `-untagged` argument.""" + issue_deprecation_warning('-untagged', None, 3) + + @staticmethod + def _handle_wikidataquery(value): + """Handle `-wikidataquery` argument.""" + issue_deprecation_warning('-wikidataquery', None, 3) + + def _handle_sparqlendpoint(self, value): + """Handle `-sparqlendpoint` argument.""" + if not value: + value = pywikibot.input('SPARQL endpoint:') + self._sparql = value + + def _handle_sparql(self, value): + """Handle `-sparql` argument.""" + if not value: + value = pywikibot.input('SPARQL query:') + return WikidataSPARQLPageGenerator( + value, site=self.site, endpoint=self._sparql) + + def _handle_mysqlquery(self, value): + """Handle `-mysqlquery` argument.""" + if not value: + value = pywikibot.input('Mysql query string:') + return MySQLPageGenerator(value, site=self.site) + + def _handle_intersect(self, value): + """Handle `-intersect` argument.""" + self.intersect = True + return True + + def _handle_subpage(self, value): + """Handle `-subpage` argument.""" + if not value: + value = pywikibot.input( + 'Maximum subpage depth:') + self.subpage_max_depth = int(value) + return True + + def _handle_logevents(self, value): + """Handle `-logevents` argument.""" + params = value.split(',') + if params[0] not in LogEntryFactory.logtypes: + raise NotImplementedError( + 'Invalid -logevents parameter "{0}"'.format(params[0])) + return self._parse_log_events(*params) + + def _old_eventlog_handler(self, logtype, value): + """Help in handling arguments LogEntryFactory.logtypes.""" + total = 500 + if value: + try: + total = int(value) + except ValueError: + params = value.split(';') + if len(params) == 2: + value, total = params + else: + value = params[0] + else: + value = None + else: + value = None + issue_deprecation_warning( + 'The usage of "{0}"'.format('-%slog' % logtype), + '-logevents:"{0}"'.format( + ','.join((logtype, value or '', str(total)))), + 4, ArgumentDeprecationWarning) + return self._parse_log_events(logtype, value, total) + def handleArg(self, arg): """Parse one argument at a time. @@ -689,11 +1166,6 @@ @return: True if the argument supplied was recognised by the factory @rtype: bool """ - def intNone(v): - """Return None if v is None or '' else return int(v).""" - return v if (v is None or v is '') else int(v) - - gen = None if not arg.startswith('-') and self._positional_arg_name: value = arg arg = '-' + self._positional_arg_name @@ -703,382 +1175,36 @@ if value == '': value = None - if arg == '-filelinks': - if not value: - value = i18n.input( - 'pywikibot-enter-file-links-processing', - fallback_prompt='Links to which file page should be ' - 'processed?') - if not value.startswith(self.site.namespace(6) + ':'): - value = 'Image:' + value - page = pywikibot.FilePage(self.site, value) - gen = FileLinksGenerator(page) - elif arg == '-linter': - if not self.site.has_extension('Linter'): - raise UnknownExtension( - '-linter needs a site with Linter extension.') - cats = self.site.siteinfo.get('linter') # Get linter categories. - valid_cats = [c for _list in cats.values() for c in _list] - - value = '' if value is None else value - cat, sep, lint_from = value.partition('/') - if not lint_from: - lint_from = None - - if cat == 'show': # Display categories of lint errors. - _i = ' ' * 4 - txt = 'Available categories of lint errors:\n' - for prio, _list in cats.items(): - txt += '{indent}{prio}\n'.format(indent=_i, prio=prio) - for c in _list: - txt += '{indent}{cat}\n'.format(indent=2 * _i, cat=c) - pywikibot.output('%s' % txt) + handler = getattr(self, '_handle_' + arg[1:], None) + if handler: + handler_result = handler(value) + if isinstance(handler_result, bool): + return handler_result + if handler_result: + self.gens.append(handler_result) return True + return False - if not cat: - lint_cats = valid_cats - elif cat in ['low', 'medium', 'high']: - lint_cats = cats[cat] - else: - lint_cats = cat.split(',') - for lint_cat in lint_cats: - if lint_cat not in valid_cats: - raise ValueError('Invalid category of lint errors: %s' - % cat) - gen = self.site.linter_pages(lint_categories='|'.join(lint_cats), - namespaces=self.namespaces, - lint_from=lint_from) - elif arg == '-unusedfiles': - gen = UnusedFilesGenerator(total=intNone(value), site=self.site) - elif arg == '-lonelypages': - gen = LonelyPagesPageGenerator(total=intNone(value), - site=self.site) - elif arg == '-unwatched': - gen = UnwatchedPagesPageGenerator(total=intNone(value), - site=self.site) - elif arg == '-property': - if not value: - question = 'Which property name to be used?' - value = pywikibot.input(question + ' (List [?])') - pnames = self.site.get_property_names() - # also use the default by <enter> key - if value in '?' or value not in pnames: - for i, item in enumerate(pnames, start=1): - pywikibot.output( - '{0:{1}}: {2}'.format(i, len(str(len(pnames))), - item)) - prefix, value = pywikibot.input_choice( - question, ListOption(self.site.get_property_names())) - gen = page_with_property_generator(value, site=self.site) - elif arg == '-usercontribs': - gen = UserContributionsGenerator(value, site=self.site) - elif arg == '-withoutinterwiki': - gen = WithoutInterwikiPageGenerator(total=intNone(value), - site=self.site) - elif arg == '-interwiki': - if not value: - value = i18n.input( - 'pywikibot-enter-page-processing', - fallback_prompt='Which page should be processed?') - page = pywikibot.Page(pywikibot.Link(value, self.site)) - gen = InterwikiPageGenerator(page) - elif arg == '-randomredirect': - # partial workaround for bug T119940 - # to use -namespace/ns with -randomredirect, -ns must be given - # before -randomredirect - # otherwise default namespace is 0 - namespaces = self.namespaces or 0 - gen = RandomRedirectPageGenerator(total=intNone(value), - site=self.site, - namespaces=namespaces) - elif arg == '-random': - # partial workaround for bug T119940 - # to use -namespace/ns with -random, -ns must be given - # before -random - # otherwise default namespace is 0 - namespaces = self.namespaces or 0 - gen = RandomPageGenerator(total=intNone(value), - site=self.site, - namespaces=namespaces) - elif arg == '-recentchanges': - rcstart = None - rcend = None - rctag = None - total = None - params = value.split(',') if value else [] - if params and not params[0].isdigit(): - rctag = params.pop(0) - if len(params) == 2: - offset = float(params[0]) - duration = float(params[1]) - if offset < 0 or duration < 0: - raise ValueError('Negative valued parameters passed.') - elif len(params) > 2: - raise ValueError('More than two parameters passed.') - else: - total = int(params[0]) if params else 60 - if len(params) == 2: - ts_time = self.site.server_time() - rcstart = ts_time + timedelta(minutes=-(offset + duration)) - rcend = ts_time + timedelta(minutes=-offset) - gen = RecentChangesPageGenerator(namespaces=self.namespaces, - total=total, - start=rcstart, - end=rcend, - site=self.site, - reverse=True, - tag=rctag, - _filter_unique=self._filter_unique) +def _old_eventlog_handler_method_factory(log_type): + """Return an old type handler method to be assigned to LogEntryFactory.""" + # Using closure is a workaround for the lack of functools.partialmethod in + # Python 2.7. + def method(self, vaule): + return self._old_eventlog_handler(log_type, vaule) + return method - elif arg == '-liverecentchanges': - self.nopreload = True - gen = LiveRCPageGenerator(site=self.site, total=intNone(value)) - elif arg == '-file': - if not value: - value = pywikibot.input('Please enter the local file name:') - gen = TextfilePageGenerator(value, site=self.site) - elif arg in ['-namespace', '-ns', '-namespaces']: - if isinstance(self._namespaces, frozenset): - warn('Cannot handle arg %s as namespaces can not ' - 'be altered after a generator is created.' - % arg, - ArgumentDeprecationWarning, 2) - return True - if not value: - value = pywikibot.input( - u'What namespace are you filtering on?') - NOT_KEY = 'not:' - if value.startswith(NOT_KEY): - value = value[len(NOT_KEY):] - resolve = self.site.namespaces.resolve - not_ns = set(resolve(value.split(','))) - if not self._namespaces: - self._namespaces = list( - set(self.site.namespaces.values()) - not_ns) - else: - self._namespaces = list( - set(resolve(self._namespaces)) - not_ns) - else: - self._namespaces += value.split(",") - return True - elif arg == '-limit': - if not value: - value = pywikibot.input('What is the limit value?') - self.limit = intNone(value) - return True - elif arg in ['-cat', '-category', '-catr', '-subcats', '-subcatsr']: - arg_dict = {'-cat': (False, CategorizedPageGenerator), - '-category': (False, CategorizedPageGenerator), - '-catr': (True, CategorizedPageGenerator), - '-subcats': (False, SubCategoriesPageGenerator), - '-subcatsr': (True, SubCategoriesPageGenerator), - } - recurse, gen_func = arg_dict[arg] - gen = self.getCategoryGen(value, recurse=recurse, gen_func=gen_func) - elif arg == '-catfilter': - cat, _ = self.getCategory(value) - self.catfilter_list.append(cat) - return True - elif arg == '-page': - if not value: - value = pywikibot.input(u'What page do you want to use?') - gen = [pywikibot.Page(pywikibot.Link(value, self.site))] - elif arg == '-pageid': - if not value: - value = pywikibot.input(u'What pageid do you want to use?') - gen = PagesFromPageidGenerator(value, site=self.site) - elif arg == '-uncatfiles': - gen = UnCategorizedImageGenerator(site=self.site) - elif arg == '-uncatcat': - gen = UnCategorizedCategoryGenerator(site=self.site) - elif arg == '-uncat': - gen = UnCategorizedPageGenerator(site=self.site) - elif arg == '-ref': - if not value: - value = pywikibot.input( - u'Links to which page should be processed?') - page = pywikibot.Page(pywikibot.Link(value, self.site)) - gen = ReferringPageGenerator(page) - elif arg == '-links': - if not value: - value = pywikibot.input( - u'Links from which page should be processed?') - page = pywikibot.Page(pywikibot.Link(value, self.site)) - gen = LinkedPageGenerator(page) - elif arg == '-weblink': - if not value: - value = pywikibot.input( - u'Pages with which weblink should be processed?') - # If url is * we make it None in order to search for every page - # with any URL. - if value == "*": - value = None - gen = LinksearchPageGenerator(value, site=self.site) - elif arg == '-transcludes': - if not value: - value = pywikibot.input( - u'Pages that transclude which page should be processed?') - page = pywikibot.Page(pywikibot.Link(value, - defaultNamespace=10, - source=self.site)) - gen = ReferringPageGenerator(page, onlyTemplateInclusion=True) - elif arg == '-start': - if not value: - value = '!' - firstpagelink = pywikibot.Link(value, self.site) - gen = self.site.allpages(start=firstpagelink.title, - namespace=firstpagelink.namespace, - filterredir=False) - elif arg == '-prefixindex': - if not value: - value = pywikibot.input( - u'What page names are you looking for?') - gen = PrefixingPageGenerator(prefix=value, site=self.site) - elif arg == '-newimages': - gen = NewimagesPageGenerator(total=intNone(value), site=self.site) - elif arg == '-newpages': - # partial workaround for bug T69249 - # to use -namespace/ns with -newpages, -ns must be given - # before -newpages - # otherwise default namespace is 0 - namespaces = self.namespaces or 0 - gen = NewpagesPageGenerator(namespaces=namespaces, - total=intNone(value), - site=self.site) - elif arg == '-unconnectedpages': - gen = self.site.unconnected_pages(total=intNone(value)) - elif arg == '-imagesused': - if not value: - value = pywikibot.input( - u'Images on which page should be processed?') - page = pywikibot.Page(pywikibot.Link(value, self.site)) - gen = ImagesPageGenerator(page) - elif arg == '-searchitem': - if not value: - value = pywikibot.input('Text to look for:') - params = value.split(':') - value = params[-1] - lang = params[0] if len(params) == 2 else None - gen = WikibaseSearchItemPageGenerator(value, language=lang, - site=self.site) - elif arg == '-search': - if not value: - value = pywikibot.input('What do you want to search for?') - # In order to be useful, all namespaces are required - gen = SearchPageGenerator(value, namespaces=[], site=self.site) - elif arg == '-google': - gen = GoogleSearchPageGenerator(value) - elif arg == '-titleregex': - if not value: - value = pywikibot.input( - 'What page names are you looking for?') - self.titlefilter_list.append(value) - return True - elif arg == '-titleregexnot': - if not value: - value = pywikibot.input( - 'All pages except which ones?') - self.titlenotfilter_list.append(value) - return True - elif arg == '-grep': - if not value: - value = pywikibot.input('Which pattern do you want to grep?') - self.articlefilter_list.append(value) - return True - elif arg == '-ql': - if not self.site.has_extension('ProofreadPage'): - raise UnknownExtension( - 'Ql filtering needs a site with ProofreadPage extension.') - value = [int(_) for _ in value.split(',')] - if min(value) < 0 or max(value) > 4: # Invalid input ql. - valid_ql = ['{0}: {1}'.format(*i) for - i in self.site.proofread_levels.items()] - valid_ql = ', '.join(valid_ql) - pywikibot.warning('Acceptable values for -ql are:\n %s' - % valid_ql) - self.qualityfilter_list = value - return True - elif arg in ('-onlyif', '-onlyifnot'): - ifnot = arg == '-onlyifnot' - if not value: - value = pywikibot.input('Which claim do you want to filter?') +for _log_type in LogEntryFactory.logtypes: + setattr( + GeneratorFactory, + '_handle_' + _log_type + 'log', + _old_eventlog_handler_method_factory(_log_type)) - p = re.compile(r'(?<!\\),') # Match "," only if there no "\" before - temp = [] # Array to store split argument - for arg in p.split(value): - temp.append(arg.replace(r'\,', ',').split('=')) - self.claimfilter_list.append((temp[0][0], temp[0][1], - dict(temp[1:]), ifnot)) - return True - elif arg == '-yahoo': - gen = YahooSearchPageGenerator(value, site=self.site) - elif arg == '-untagged': - issue_deprecation_warning(arg, None, 2) - elif arg == '-wikidataquery': - issue_deprecation_warning(arg, None, 2) - elif arg == '-sparqlendpoint': - if not value: - value = pywikibot.input('SPARQL endpoint:') - self._sparql = value - elif arg == '-sparql': - if not value: - value = pywikibot.input('SPARQL query:') - gen = WikidataSPARQLPageGenerator(value, site=self.site, endpoint=self._sparql) - elif arg == '-mysqlquery': - if not value: - value = pywikibot.input('Mysql query string:') - gen = MySQLPageGenerator(value, site=self.site) - elif arg == '-intersect': - self.intersect = True - return True - elif arg == '-subpage': - if not value: - value = pywikibot.input( - 'Maximum subpage depth:') - self.subpage_max_depth = int(value) - return True - elif arg == '-logevents': - params = value.split(',') - if params[0] not in LogEntryFactory.logtypes: - raise NotImplementedError( - 'Invalid -logevents parameter "{0}"'.format(params[0])) - gen = self._parse_log_events(*params) - elif arg.startswith('-'): - mode, log, tail = arg.partition('log') - # exclude -log, -nolog - if log == 'log' and mode not in ['-', '-no'] and not tail: - logtype = mode[1:] - if logtype not in LogEntryFactory.logtypes: - raise NotImplementedError( - 'Invalid logevent option "{0}log"'.format(mode)) - total = 500 - if value: - try: - total = int(value) - except ValueError: - params = value.split(';') - if len(params) == 2: - value, total = params - else: - value = params[0] - else: - value = None - else: - value = None - issue_deprecation_warning( - 'The usage of "{0}"'.format(arg), - '-logevents:"{0}"'.format( - ','.join((logtype, value or '', str(total)))), - 2, ArgumentDeprecationWarning) - gen = self._parse_log_events(logtype, value, total) - if gen: - self.gens.append(gen) - return True - else: - return False +def _int_none(v): + """Return None if v is None or '' else return int(v).""" + return v if (v is None or v is '') else int(v) @deprecated('Site.allpages()') diff --git a/tests/pagegenerators_tests.py b/tests/pagegenerators_tests.py index db7a33c..bcc7afc 100755 --- a/tests/pagegenerators_tests.py +++ b/tests/pagegenerators_tests.py @@ -25,7 +25,7 @@ from pywikibot.tools import has_module, suppress_warnings -from tests import join_data_path +from tests import join_data_path, mock from tests.aspects import ( unittest, TestCase, @@ -1212,8 +1212,9 @@ """Test -onlyif without qualifiers.""" gf = pagegenerators.GeneratorFactory(site=self.site) gf.handleArg('-page:Q15745378') - gf.handleArg('-onlyif:P1476=International Journal of Minerals\\, ' - 'Metallurgy\\, and Materials') + self.assertTrue(gf.handleArg( + '-onlyif:P1476=International Journal of Minerals\\, ' + 'Metallurgy\\, and Materials')) gen = gf.getCombinedGenerator() self.assertIsNotNone(gen) self.assertEqual(len(set(gen)), 1) @@ -1299,12 +1300,16 @@ def test_logevents_parse(self): """Test wrong logevents option.""" - gf = pagegenerators.GeneratorFactory() + factory = pagegenerators.GeneratorFactory + gf = factory() self.assertFalse(gf.handleArg("-log")) self.assertFalse(gf.handleArg("-log:text_here")) self.assertRaises(NotImplementedError, gf.handleArg, '-logevents:anyevent') - self.assertRaises(NotImplementedError, gf.handleArg, '-anotherlog') + # test that old format log option is not handled by any handler method. + gf_mock = mock.create_autospec(gf) + self.assertFalse(factory.handleArg(gf_mock, '-anotherlog')) + self.assertFalse(gf_mock.method_calls) def test_logevents_default(self): """Test old logevents option handling.""" -- To view, visit https://gerrit.wikimedia.org/r/440096 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I844c781d9bc80c30734d2f82def24db8633b7f0d Gerrit-Change-Number: 440096 Gerrit-PatchSet: 10 Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot Gerrit-CC: Dvorapa <dvorapa(a)seznam.cz>

5 years, 10 months

[Gerrit] pywikibot/core[master]: docs/conf.py: Set the `default_role` to 'code'

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/440160 ) Change subject: docs/conf.py: Set the `default_role` to 'code' ...................................................................... docs/conf.py: Set the `default_role` to 'code' The "default default role" is :title-reference:[1] which is currently redered to <cite> tags. Having a look at "git grep '`' -- '*.py'" results, it seems to more appropriate to have all the current usage of '`' interpreted as code rather than title. [1]: http://docutils.sourceforge.net/docs/ref/rst/roles.html Change-Id: Ic1341dd4463379a760a7a735f519bcd0a3b67858 --- M docs/conf.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/docs/conf.py b/docs/conf.py index 44e6b7b..e6e4909 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -88,7 +88,7 @@ exclude_patterns = ['_build'] # The reST default role (used for this markup: `text`) to use for all documents. -# default_role = None +default_role = 'code' # If true, '()' will be appended to :func: etc. cross-reference text. # add_function_parentheses = True -- To view, visit https://gerrit.wikimedia.org/r/440160 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic1341dd4463379a760a7a735f519bcd0a3b67858 Gerrit-Change-Number: 440160 Gerrit-PatchSet: 2 Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

[Gerrit] pywikibot/core[master]: shell.py: call `sys.__interactivehook__()` if available

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/436820 ) Change subject: shell.py: call `sys.__interactivehook__()` if available ...................................................................... shell.py: call `sys.__interactivehook__()` if available This pactch makes shell.py acts more similar to the built-in interactive shell, where Python native code calls the hook. The hook may be user-defined, or be the default implementation defined by the site module of CPython 3.4+ Standard Library. The site module is automatically imported if -S is not specified to the Python interpreter. The default implementation of the hook does all the magic of loading readline, saving history, and tab-completion. Unless the user defined their own sys.__interactivehook__, which may include other features and/or exclude readline, and which we honor in this patch, by default, this patch only works in CPython 3.4 or newer, after the fix of Python issue #5845 [1], and only on Python installs where GNU readline is compiled into the CPython interpreter binary. Other Python implementations, such as IronPython, may also include their implementations of the hook, and we try our best to simulate their built-in shell, but it is not our main concern. * In Windows, Python uses the ReadConsoleW() Windows API, which has some cmd-like history functionality built-in. It is able to use the history of a previous invokation of Python as long as the terminal window stays open. * In older versions of Python, sys.__interactivehook__ is not defined by site. * In newer versions of Python, but without readline compiled, interactivehook will attempt to import readline but silently fail. In these cases, you get the plain old interactive prompt, as if this patch never existed. In any case, this patch shall attempt to make the behavior of shell.py closer to what would be provided if one invokes the built-in Python interactive shell directly. [1] https://bugs.python.org/issue5845 Bug: T196168 Change-Id: I410c9717370cff70fa441f06500625956e8596fc --- M scripts/shell.py 1 file changed, 9 insertions(+), 3 deletions(-) Approvals: jenkins-bot: Verified Dvorapa: Looks good to me, but someone else must approve Dalba: Looks good to me, approved diff --git a/scripts/shell.py b/scripts/shell.py index b119a6c..5333556 100755 --- a/scripts/shell.py +++ b/scripts/shell.py @@ -20,6 +20,9 @@ # from __future__ import absolute_import, print_function, unicode_literals +import code +import sys + def main(*args): """Script entry point.""" @@ -38,12 +41,16 @@ print('{} arguments: {}\n' # noqa: T001 .format(warn_type, ', '.join(args))) - import code + # Various stuffs in Python 3.4+, such as history file. + # This is defined in the site module of the Python Standard Library, + # and usually called by the built-in CPython interactive shell. + if hasattr(sys, '__interactivehook__'): + sys.__interactivehook__() + code.interact("""Welcome to the Pywikibot interactive shell!""", local=env) if __name__ == "__main__": - import sys if sys.platform == 'win32': import os os.system('title Python {} Shell'.format(*sys.version.split(' ', 1))) @@ -51,5 +58,4 @@ args = [] if sys.argv and sys.argv[0].endswith(('shell', 'shell.py')): args = sys.argv[1:] - del sys main(*args) -- To view, visit https://gerrit.wikimedia.org/r/436820 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I410c9717370cff70fa441f06500625956e8596fc Gerrit-Change-Number: 436820 Gerrit-PatchSet: 6 Gerrit-Owner: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

[Gerrit] pywikibot/core[master]: [bugfix] Remove multiple empty sections at once

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/437227 ) Change subject: [bugfix] Remove multiple empty sections at once ...................................................................... [bugfix] Remove multiple empty sections at once + Add textlib method to parse sections Bug: T196324 Change-Id: Ic011f8be6ee64572e5b88a551ce18405d36c214d --- M pywikibot/cosmetic_changes.py M pywikibot/textlib.py M tests/cosmetic_changes_tests.py 3 files changed, 184 insertions(+), 42 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/cosmetic_changes.py b/pywikibot/cosmetic_changes.py index a01d7f2..c33cdd1 100755 --- a/pywikibot/cosmetic_changes.py +++ b/pywikibot/cosmetic_changes.py @@ -647,11 +647,13 @@ def removeEmptySections(self, text): """Cleanup empty sections.""" - exceptions = ['comment', 'pre', 'source', 'nowiki', 'code', - 'startspace'] - - skippings = ['comment'] + # comments, categories, and interwikis + skippings = ['comment', 'category', 'interwiki'] skip_regexes = _get_regexes(skippings, self.site) + # we want only interwikis, not interlanguage links + skip_regexes[1] = re.compile( + skip_regexes[1].pattern.replace(':?', '')) + # site defined templates skip_templates = { 'cs': ('Pahýl[ _]část',), # stub section } @@ -659,25 +661,33 @@ for template in skip_templates[self.site.code]: skip_regexes.append( re.compile(r'\{\{\s*%s\s*\}\}' % template, re.I)) + # empty lists + skip_regexes.append(re.compile(r'(?m)^[\*#] *$')) + # get stripped sections stripped_text = text for reg in skip_regexes: stripped_text = reg.sub(r'', stripped_text) + strip_sections = textlib.extract_sections( + stripped_text, self.site)[1] - stripped_pattern = re.compile( - r'\n((=+) *[^\n=]+? *\2) *\n\s*(?=(\2 *[^\n=]+? *\2))') - pos = 0 - while True: - match = stripped_pattern.search(stripped_text[pos:]) - if not match: - break - pattern = re.compile(r'\n{}.+?(?={})'.format( - match.group(1), match.group(3)), re.DOTALL) - text = textlib.replaceExcept(text, pattern, r'\n', - exceptions=exceptions) - pos = match.end() + # get proper sections + header, sections, footer = textlib.extract_sections(text, self.site) - return text + # iterate stripped sections and create a new page body + new_body = [] + for i, strip_section in enumerate(strip_sections): + current_heading = sections[i][0] + try: + next_heading = sections[i + 1][0] + except IndexError: + next_heading = '' + current_dep = (len(current_heading) + - len(current_heading.lstrip('='))) + next_dep = len(next_heading) - len(next_heading.lstrip('=')) + if strip_section[1].strip() or current_dep < next_dep: + new_body = new_body + list(sections[i]) + return header + ''.join(new_body) + footer def removeUselessSpaces(self, text): """Cleanup multiple or trailing spaces.""" diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py index e295783..e0468ae 100644 --- a/pywikibot/textlib.py +++ b/pywikibot/textlib.py @@ -37,6 +37,7 @@ unicode = str else: from HTMLParser import HTMLParser + from itertools import izip as zip try: import mwparserfromhell @@ -820,6 +821,99 @@ return text +# ------------------------------- +# Functions dealing with sections +# ------------------------------- +def extract_sections(text, site=None): + """ + Return section headings and contents found in text. + + @return: The returned tuple contains the text parsed into three + parts: The first part is a string containing header part above + the first heading. The last part is also a string containing + footer part after the last section. The middle part is a list + of tuples, each tuple containing a string with section heading + and a string with section content. Example article:: + + '''A''' is a thing. + + == History of A == + Some history... + + == Usage of A == + Some usage... + + [[Category:Things starting with A]] + + ...is parsed into the following tuple:: + + (header, body, footer) + header = "'''A''' is a thing." + body = [('== History of A ==', 'Some history...'), + ('== Usage of A ==', 'Some usage...')] + footer = '[[Category:Things starting with A]]' + + @rtype: tuple of (str, list of tuples, str) + """ + headings = [] + contents = [] + body = [] + + # Find valid headings + heading_regex = _get_regexes(['header'], site)[0] + pos = 0 + while True: + match = heading_regex.search(text[pos:]) + if not match: + break + start = pos + match.start() + end = pos + match.end() + if not (isDisabled(text, start) + or isDisabled(text, end)): + headings += [(match.group(), start, end)] + pos = end + + if headings: + # Assign them their contents + for i, current in enumerate(headings): + try: + following = headings[i + 1] + except IndexError: + following = None + if following: + contents.append(text[current[2]:following[1]]) + else: + contents.append(text[current[2]:]) + body = [(heading[0], section) + for heading, section in zip(headings, contents)] + + # Find header and footer contents + header = text[:headings[0][1]] if headings else text + + last_section = body[-1][1] if body else header + skippings = ['category', 'interwiki'] + footer_regexes = _get_regexes(skippings, site) + # we want only interwikis, not interlanguage links + footer_regexes[1] = re.compile( + footer_regexes[1].pattern.replace(':?', '')) + # find where to cut + positions = [] + for reg in footer_regexes: + match = reg.search(last_section) + if match: + positions.append(match.start()) + pos = min(pos for pos in positions) if positions else len(last_section) + + # Strip footer from last section content + last_section, footer = last_section[:pos], last_section[pos:] + if body: + body[-1] = (body[-1][0], last_section) + else: + header = last_section + + return header, body, footer + + # ----------------------------------------------- # Functions dealing with interwiki language links # ----------------------------------------------- diff --git a/tests/cosmetic_changes_tests.py b/tests/cosmetic_changes_tests.py index 7d8a6a2..b1c6361 100644 --- a/tests/cosmetic_changes_tests.py +++ b/tests/cosmetic_changes_tests.py @@ -65,31 +65,6 @@ '<code> </code>', self.cct.resolveHtmlEntities('<code> </code>')) - def test_removeEmptySections(self): - """Test removeEmptySections method.""" - # same level - self.assertEqual( - '\n==Bar==', - self.cct.removeEmptySections('\n== Foo ==\n\n==Bar==')) - # different level - self.assertEqual( - '\n===Foo===\n\n==Bar==', - self.cct.removeEmptySections('\n===Foo===\n\n==Bar==')) - self.assertEqual( - '\n==Foo==\n\n===Bar===', - self.cct.removeEmptySections('\n==Foo==\n\n===Bar===')) - # comment inside - self.assertEqual( - '\n==Bar==', - self.cct.removeEmptySections('\n==Foo==\n\n==Bar==')) - # comments and content between - testcase = '\n== Foo ==\n\nBaz\n\n== Bar ==' - self.assertEqual(testcase, self.cct.removeEmptySections(testcase)) - # inside comment - self.assertEqual( - '', - self.cct.removeEmptySections('')) - def test_removeUselessSpaces(self): """Test removeUselessSpaces method.""" self.assertEqual('Foo bar', @@ -294,6 +269,69 @@ """Test cosmetic_changes requiring a live wiki.""" + def test_removeEmptySections(self): + """Test removeEmptySections method.""" + content = '\nSome content' + # same level + self.assertEqual( + '\n==Bar==' + content, + self.cct.removeEmptySections('\n== Foo ==\n\n==Bar==' + content)) + # different level + self.assertEqual( + '\n==Bar==' + content, + self.cct.removeEmptySections('\n===Foo===\n\n==Bar==' + content)) + testcase = '\n==Foo==\n\n===Bar===' + content + self.assertEqual(testcase, self.cct.removeEmptySections(testcase)) + # multiple empty sections + self.assertEqual( + '\n==Baz==' + content, + self.cct.removeEmptySections('\n==Foo==\n==Bar==\n==Baz==' + + content)) + # comment inside + self.assertEqual( + '\n==Bar==' + content, + self.cct.removeEmptySections('\n==Foo==\n\n==Bar==' + + content)) + # comments and content between + testcase = ('\n== Foo ==\n\nBaz\n\n== Bar ==' + + content) + self.assertEqual(testcase, self.cct.removeEmptySections(testcase)) + # inside comment + testcase = '' + content + self.assertEqual(testcase, self.cct.removeEmptySections(testcase)) + testcase = '\n==Foo==\n' + content + self.assertEqual(testcase, self.cct.removeEmptySections(testcase)) + testcase = '\n==Bar==' + content + self.assertEqual(testcase, self.cct.removeEmptySections(testcase)) + # empty list item + self.assertEqual( + '\n==Baz==' + content, + self.cct.removeEmptySections('\n==Foo==\n*\n==Bar==\n#\n==Baz==' + + content)) + self.assertEqual( + '\n==Baz==' + content, + self.cct.removeEmptySections('\n==Foo==\n* \n==Baz==' + + content)) + testcase = '\n==Foo==\n* item\n==Bar==' + content + self.assertEqual(testcase, self.cct.removeEmptySections(testcase)) + # empty first section + self.assertEqual( + '==Bar==' + content, + self.cct.removeEmptySections('==Foo==\n==Bar==' + content)) + # empty last section + self.assertEqual( + '\n[[Category:Baz]]', + self.cct.removeEmptySections('\n==Bar==\n[[Category:Baz]]')) + # complicated + self.assertEqual( + '\n[[Category:Baz]]', + self.cct.removeEmptySections('\n==Bar==\n* ' + '\n[[Category:Baz]]')) + self.assertEqual( + '\n[[cs:Foo]]\n[[Category:Baz]]', + self.cct.removeEmptySections('\n==Bar==\n[[cs:Foo]]' + '\n[[Category:Baz]]')) + def test_translateAndCapitalizeNamespaces(self): """Test translateAndCapitalizeNamespaces method.""" self.assertEqual( -- To view, visit https://gerrit.wikimedia.org/r/437227 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic011f8be6ee64572e5b88a551ce18405d36c214d Gerrit-Change-Number: 437227 Gerrit-PatchSet: 11 Gerrit-Owner: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Rachmat.Wahidi <rachmatwahidi.site(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

[Gerrit] pywikibot/core[master]: [bugfix] Fix stub template position and above interwiki comments

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/437175 ) Change subject: [bugfix] Fix stub template position and above interwiki comments ...................................................................... [bugfix] Fix stub template position and above interwiki comments Per T57034 Bug: T57034 Change-Id: I276f5fbdacdb13404bc1c63aefc0d2a789cc4f67 --- M pywikibot/textlib.py 1 file changed, 12 insertions(+), 0 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py index e295783..dd73fe6 100644 --- a/pywikibot/textlib.py +++ b/pywikibot/textlib.py @@ -1076,6 +1076,10 @@ r'alphabetically by name\)-->') above_interwiki.append(comment) + if site.family.name == 'wikipedia' and site.code in ('ba', 'crh', 'krc'): + comment = re.compile(r'') + above_interwiki.append(comment) + if above_interwiki: interwiki = _get_regexes(['interwiki'], site)[0] first_interwiki = interwiki.search(newtext) @@ -1373,6 +1377,14 @@ re.I | re.DOTALL) under_categories.append(personendaten) + if site.sitename == 'wikipedia:yi': + stub = re.compile(r'\{\{.*?שטומף *\}\}', re.I) + under_categories.append(stub) + + if site.family.name == 'wikipedia' and site.code in ('simple', 'en'): + stub = re.compile(r'\{\{.*?stub *\}\}', re.I) + under_categories.append(stub) + if under_categories: category = _get_regexes(['category'], site)[0] for last_category in category.finditer(newtext): -- To view, visit https://gerrit.wikimedia.org/r/437175 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I276f5fbdacdb13404bc1c63aefc0d2a789cc4f67 Gerrit-Change-Number: 437175 Gerrit-PatchSet: 2 Gerrit-Owner: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Cyberpower678 <cybernet678(a)yahoo.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm(a)member.fsf.org> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot

5 years, 10 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits June 2018