Pywikibot-commits August 2018

pywikibot-commits@lists.wikimedia.org

1 participants
37 discussions

[Gerrit] pywikibot/core[master]: Introduce a new APISite property: mw_version
by jenkins-bot (Code Review) 31 Aug '18

31 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/455505 ) Change subject: Introduce a new APISite property: mw_version ...................................................................... Introduce a new APISite property: mw_version APISite.version() is a widely used method. In most of its usages the version string is converted to a MediaWikiVersion object. To improve readability and ease of use, introduce a new property, mw_version, which returns a MediaWikiVersion object and caches the result. Use the new property throughout the codebase. Deprecate pywikibot.MediaWikiVersion in favour of pywikibot.tools.MediaWikiVersion. Change-Id: I75de443b24d2eb478724c416db8f2d734b6bb806 --- M pywikibot/__init__.py M pywikibot/cosmetic_changes.py M pywikibot/data/api.py M pywikibot/page.py M pywikibot/pagegenerators.py M pywikibot/site.py M tests/api_tests.py M tests/edit_tests.py M tests/logentry_tests.py M tests/page_tests.py M tests/paraminfo_tests.py M tests/site_tests.py M tests/wikibase_edit_tests.py M tests/wikibase_tests.py 14 files changed, 110 insertions(+), 113 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py index f850057..2d66ece 100644 --- a/pywikibot/__init__.py +++ b/pywikibot/__init__.py @@ -63,7 +63,7 @@ deprecated as __deprecated, deprecate_arg as _deprecate_arg, normalize_username, - MediaWikiVersion, + MediaWikiVersion as _MediaWikiVersion, redirect_func, ModuleDeprecationWrapper as _ModuleDeprecationWrapper, PY2, @@ -740,8 +740,7 @@ "WbQuantity now expects a 'site' parameter. This is needed to " "ensure correct handling of error bounds.") return False - return MediaWikiVersion( - site.version()) < MediaWikiVersion('1.29.0-wmf.2') + return site.mw_version < '1.29.0-wmf.2' @staticmethod def _todecimal(value): @@ -1457,3 +1456,8 @@ warning_message='pywikibot.UploadWarning is deprecated; ' 'use APISite.upload with a warning handler instead.', since='20150921') +wrapper._add_deprecated_attr( + 'MediaWikiVersion', _MediaWikiVersion, + warning_message='pywikibot.MediaWikiVersion is deprecated; ' + 'use pywikibot.tools.MediaWikiVersion instead.', + since='20180827') diff --git a/pywikibot/cosmetic_changes.py b/pywikibot/cosmetic_changes.py index ac4d548..318515a 100755 --- a/pywikibot/cosmetic_changes.py +++ b/pywikibot/cosmetic_changes.py @@ -74,7 +74,6 @@ from pywikibot.textlib import (_MultiTemplateMatchBuilder, FILE_LINK_REGEX, _get_regexes) from pywikibot.tools import deprecated_args, first_lower, first_upper -from pywikibot.tools import MediaWikiVersion # Subpage templates. Must be in lower case, @@ -401,8 +400,8 @@ # a clone is needed. Won't change the namespace dict namespaces = list(namespace) if namespace == 6 and self.site.family.name == 'wikipedia': - if self.site.code in ('en', 'fr') and MediaWikiVersion( - self.site.version()) >= MediaWikiVersion('1.14'): + if self.site.code in ('en', 'fr') \ + and self.site.mw_version >= '1.14': # do not change "Image" on en-wiki and fr-wiki assert u'Image' in namespaces namespaces.remove(u'Image') diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py index 9adc8f5..f10e838 100644 --- a/pywikibot/data/api.py +++ b/pywikibot/data/api.py @@ -249,23 +249,22 @@ assert ('query' in self._modules) is ('main' in self._paraminfo) if 'query' in self._modules: return - _mw_ver = MediaWikiVersion(self.site.version()) + mw_ver = self.site.mw_version - if _mw_ver < MediaWikiVersion('1.15'): - self._parse_help(_mw_ver) + if mw_ver < '1.15': + self._parse_help(mw_ver) # The paraminfo api deprecated the old request syntax of # querymodules='info'; to avoid warnings sites with 1.25wmf4+ # must only use 'modules' parameter. if self.modules_only_mode is None: - self.modules_only_mode = _mw_ver >= MediaWikiVersion('1.25wmf4') + self.modules_only_mode = mw_ver >= '1.25wmf4' if self.modules_only_mode: self.paraminfo_keys = frozenset(['modules']) # v1.18 and earlier paraminfo doesnt include modules; must use 'query' # Assume that by v1.26, it will be desirable to prefetch 'query' - if _mw_ver > MediaWikiVersion('1.26') \ - or _mw_ver < MediaWikiVersion('1.19'): + if mw_ver > '1.26' or mw_ver < '1.19': self.preloaded_modules |= {'query'} self._fetch(self.preloaded_modules) @@ -593,7 +592,7 @@ assert 'query' in self._modules or 'paraminfo' not in self._paraminfo - if MediaWikiVersion(self.site.version()) < MediaWikiVersion("1.12"): + if self.site.mw_version < '1.12': # When the help is parsed, all paraminfo should already be loaded # and the caller is responsible for detecting missing modules. pywikibot.log('ParamInfo did not detect modules: %s' @@ -691,7 +690,7 @@ parameters = self._paraminfo[module].get('parameters', []) submodules = set() # Advanced submodule into added to MW API in df80f1ea - if self.site.version() >= MediaWikiVersion('1.26wmf9'): + if self.site.mw_version >= '1.26wmf9': # This is supplying submodules even if they aren't submodules # of the given module so skip those for param in parameters: @@ -708,7 +707,7 @@ submodules.add(child) else: # Boolean submodule info added to MW API in afa153ae - if self.site.version() < MediaWikiVersion('1.24wmf18'): + if self.site.mw_version < '1.24wmf18': if module == 'main': params = {'action'} elif module == 'query': @@ -1458,8 +1457,7 @@ # otherwise be a problem. # This situation is only tripped when one of the first actions # on the site is a write action and the extension isn't installed. - if (self.write and MediaWikiVersion( - self.site.version()) >= MediaWikiVersion('1.23') + if (self.write and self.site.mw_version >= '1.23' or self.action == 'edit' and self.site.has_extension('AssertEdit')): pywikibot.debug(u"Adding user assertion", _logger) @@ -2622,7 +2620,7 @@ % self.__class__.__name__) parameters['indexpageids'] = True # always ask for list of pageids - if MediaWikiVersion(self.site.version()) < MediaWikiVersion('1.21'): + if self.site.mw_version < '1.21': self.continue_name = 'query-continue' self.continue_update = self._query_continue else: @@ -3225,7 +3223,7 @@ login_request['lgdomain'] = self.site.family.ldapDomain # get token using meta=tokens if supported - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion('1.27'): + if self.site.mw_version >= '1.27': login_request["lgtoken"] = self.get_login_token() self.site._loginstatus = -2 # IN_PROGRESS @@ -3262,7 +3260,7 @@ @return: login token @rtype: str """ - if MediaWikiVersion(self.site.version()) < MediaWikiVersion('1.27'): + if self.site.mw_version < '1.27': raise NotImplementedError('The method get_login_token() requires ' 'at least MediaWiki version 1.27.') login_token_request = self.site._request( diff --git a/pywikibot/page.py b/pywikibot/page.py index 58c9596..611320c 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -43,7 +43,7 @@ from pywikibot import textlib from pywikibot.tools import ( compute_file_hash, - MediaWikiVersion, UnicodeMixin, ComparableMixin, DotReadableDict, + UnicodeMixin, ComparableMixin, DotReadableDict, deprecated, deprecate_arg, deprecated_args, issue_deprecation_warning, add_full_name, manage_wrapping, ModuleDeprecationWrapper as _ModuleDeprecationWrapper, PY2, @@ -1110,7 +1110,7 @@ @rtype: set """ # New API since commit 32083235eb332c419df2063cf966b3400be7ee8a - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion('1.25wmf14'): + if self.site.mw_version >= '1.25wmf14': self.site.loadpageinfo(self) return self._applicable_protections diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index 7378a96..7be9948 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -55,7 +55,6 @@ UnknownExtension, ) from pywikibot.proofreadpage import ProofreadPage -from pywikibot.tools import MediaWikiVersion if sys.version_info[0] > 2: basestring = (str, ) @@ -1479,7 +1478,7 @@ 'content': content, 'namespaces': namespaces, } if start: - if category.site.version() < MediaWikiVersion('1.18'): + if category.site.mw_version < '1.18': kwargs.pop('total', None) count = 0 for article in category.articles(**kwargs): diff --git a/pywikibot/site.py b/pywikibot/site.py index 0177c0e..b8ec8b4 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -2488,7 +2488,7 @@ raise NotImplementedError( 'MediaWiki messages missing: {0}'.format(needed_mw_messages)) - if MediaWikiVersion(self.version()) < MediaWikiVersion('1.16'): + if self.mw_version < '1.16': for key, value in msgs.items(): if key == 'and' and value == ', and': # v1.14 defined and as ', and'; fixed in v1.15 @@ -2527,7 +2527,7 @@ req['title'] = title if includecomments is True: req['includecomments'] = u'' - if MediaWikiVersion(self.version()) > MediaWikiVersion("1.24wmf7"): + if self.mw_version > '1.24wmf7': key = 'wikitext' req['prop'] = key else: @@ -2563,7 +2563,7 @@ @return: the current server time @rtype: L{Timestamp} """ - if MediaWikiVersion(self.version()) >= MediaWikiVersion("1.16"): + if self.mw_version >= '1.16': return pywikibot.Timestamp.fromISOformat( self.siteinfo.get('time', expiry=0)) else: @@ -2634,7 +2634,7 @@ # and Image became File with Image as an alias. # For versions lower than 1.14, APISite needs to override # the defaults defined in Namespace. - is_mw114 = MediaWikiVersion(self.version()) >= MediaWikiVersion('1.14') + is_mw114 = self.mw_version >= '1.14' for nsdata in self.siteinfo.get('namespaces', cache=False).values(): ns = nsdata.pop('id') @@ -2757,6 +2757,19 @@ return version @property + def mw_version(self): + """Return self.version() as a MediaWikiVersion object. + + Cache the result for 24 hours. + @rtype: MediaWikiVersion + """ + mw_ver, cache_time = getattr(self, '_mw_version_time', (None, None)) + if mw_ver is None or time.time() - cache_time > 60 * 60 * 24: + mw_ver = MediaWikiVersion(self.version()) + setattr(self, '_mw_version_time', (mw_ver, time.time())) + return mw_ver + + @property def has_image_repository(self): """Return True if site has a shared image repository like Commons.""" code, fam = self.shared_image_repository() @@ -3374,7 +3387,7 @@ else: rvgen.request['titles'] = list(cache.keys()) rvgen.request['rvprop'] = rvprop - if self.version() >= MediaWikiVersion('1.32'): + if self.mw_version >= '1.32': rvgen.request['rvslots'] = '*' pywikibot.output(u"Retrieving %s pages from %s." % (len(cache), self)) @@ -3426,19 +3439,19 @@ Valid tokens depend on mw version. """ - _version = MediaWikiVersion(self.version()) - if _version < MediaWikiVersion('1.20'): + mw_ver = self.mw_version + if mw_ver < '1.20': types_wiki = self._paraminfo.parameter('query+info', 'token')['type'] types_wiki.append('patrol') valid_types = [token for token in types if token in types_wiki] # Pre 1.17, preload token was the same as the edit token. - if _version < MediaWikiVersion('1.17'): + if mw_ver < '1.17': if 'patrol' in types and 'edit' not in valid_types: valid_types.append('edit') - elif _version < MediaWikiVersion('1.24wmf19'): + elif mw_ver < '1.24wmf19': types_wiki = self._paraminfo.parameter('tokens', 'type')['type'] valid_types = [token for token in types if token in types_wiki] @@ -3496,8 +3509,8 @@ r'Action \'\w+\' is not allowed for the current user', text) user_tokens = {} - _version = MediaWikiVersion(self.version()) - if _version < MediaWikiVersion('1.20'): + mw_ver = self.mw_version + if mw_ver < '1.20': if all: types_wiki = self._paraminfo.parameter('query+info', 'token')['type'] @@ -3521,8 +3534,7 @@ # patrol token require special handling. # TODO: try to catch exceptions? if 'patrol' in valid_tokens: - if MediaWikiVersion( - '1.14') <= _version < MediaWikiVersion('1.17'): + if '1.14' <= mw_ver < '1.17': if 'edit' in user_tokens: user_tokens['patrol'] = user_tokens['edit'] else: @@ -3541,7 +3553,7 @@ if 'patroltoken' in item: user_tokens['patrol'] = item['patroltoken'] else: - if _version < MediaWikiVersion('1.24wmf19'): + if mw_ver < '1.24wmf19': if all is not False: types_wiki = self._paraminfo.parameter('tokens', 'type')['type'] @@ -3903,9 +3915,7 @@ if isinstance(member_type, basestring): member_type = {member_type} - if (member_type and - (sortby == 'timestamp' or - MediaWikiVersion(self.version()) < MediaWikiVersion("1.12"))): + if member_type and (sortby == 'timestamp' or self.mw_version < '1.12'): # Retrofit cmtype/member_type, available on MW API 1.12+, # to use namespaces available on earlier versions. @@ -3963,7 +3973,7 @@ raise ValueError("categorymembers: " "invalid combination of 'sortby' and 'endtime'") if startprefix and sortby != 'timestamp': - if self.version() < MediaWikiVersion('1.18'): + if self.mw_version < '1.18': raise NotImplementedError( 'categorymembers: "startprefix" requires MW 1.18+') cmargs['gcmstartsortkeyprefix'] = startprefix @@ -3976,7 +3986,7 @@ raise ValueError("categorymembers: " "invalid combination of 'sortby' and 'startsort'") if endprefix and sortby != 'timestamp': - if self.version() < MediaWikiVersion('1.18'): + if self.mw_version < '1.18': raise NotImplementedError( 'categorymembers: "endprefix" requires MW 1.18+') cmargs['cmendsortkeyprefix'] = endprefix @@ -4069,13 +4079,13 @@ rvargs = {'type_arg': 'info|revisions'} rvargs['rvprop'] = ['ids', 'timestamp', 'flags', 'comment', 'user'] - if self.version() >= MediaWikiVersion('1.32'): + if self.mw_version >= '1.32': rvargs['rvslots'] = '*' # 'roles' is not implemented in Revision class yet. # rvargs['rvprop'].append('roles') - if MediaWikiVersion(self.version()) >= MediaWikiVersion('1.21'): + if self.mw_version >= '1.21': rvargs['rvprop'].append('contentmodel') - if MediaWikiVersion(self.version()) >= MediaWikiVersion('1.19'): + if self.mw_version >= '1.19': rvargs['rvprop'].append('sha1') if content: rvargs['rvprop'].append('content') @@ -4489,7 +4499,7 @@ total=total) bkgen.request['bkprop'] = ['id', 'user', 'by', 'timestamp', 'expiry', 'reason', 'range', 'flags'] - if MediaWikiVersion(self.version()) >= MediaWikiVersion('1.18'): + if self.mw_version >= '1.18': bkgen.request['bkprop'] += ['userid'] if starttime: bkgen.request["bkstart"] = starttime @@ -4729,7 +4739,7 @@ if reverse: rcgen.request["rcdir"] = "newer" if pagelist: - if MediaWikiVersion(self.version()) > MediaWikiVersion("1.14"): + if self.mw_version > '1.14': pywikibot.warning( u"recentchanges: pagelist option is disabled; ignoring.") else: @@ -4783,7 +4793,7 @@ type such as NoneType or bool """ where_types = ['text', 'title', 'titles'] - if MediaWikiVersion(self.version()) >= MediaWikiVersion('1.17'): + if self.mw_version >= '1.17': where_types.append('nearmatch') if not searchstring: raise Error("search: searchstring cannot be empty") @@ -4803,7 +4813,7 @@ issue_deprecation_warning("where='titles'", "where='title'", 2, since='20160224') - if MediaWikiVersion(self.version()) < MediaWikiVersion('1.11'): + if self.mw_version < '1.11': where = 'titles' else: where = 'title' @@ -4813,7 +4823,7 @@ gsrsearch=searchstring, gsrwhat=where, namespaces=namespaces, total=total, g_content=content) - if MediaWikiVersion(self.version()) < MediaWikiVersion('1.23'): + if self.mw_version < '1.23': srgen.request['gsrredirects'] = get_redirects return srgen @@ -4987,9 +4997,9 @@ """ usprop = ['blockinfo', 'groups', 'editcount', 'registration', 'emailable'] - if MediaWikiVersion(self.version()) >= MediaWikiVersion('1.16'): + if self.mw_version >= '1.16': usprop.append('gender') - if MediaWikiVersion(self.version()) >= MediaWikiVersion('1.17'): + if self.mw_version >= '1.17': usprop.append('rights') usgen = api.ListGenerator( 'users', site=self, parameters={ @@ -5044,11 +5054,11 @@ redirects = mapping[redirects] params = {} if redirects is not None: - if MediaWikiVersion(self.version()) < MediaWikiVersion('1.26'): + if self.mw_version < '1.26': if redirects == 'all': warn("parameter redirects=None to retrieve 'all' random" 'page types is not supported by mw version {0}. ' - 'Using default.'.format(self.version()), + 'Using default.'.format(self.mw_version), UserWarning) params['grnredirect'] = redirects == 'redirects' else: @@ -5182,7 +5192,7 @@ watch_items = {'watch', 'unwatch', 'preferences', 'nochange'} if watch in watch_items: - if MediaWikiVersion(self.version()) < MediaWikiVersion("1.16"): + if self.mw_version < '1.16': if watch in ['preferences', 'nochange']: pywikibot.warning(u'The watch value {0} is not supported ' 'by {1}'.format(watch, self)) @@ -5329,8 +5339,7 @@ @type reason: str """ # Check wiki version to see if action=mergehistory is supported - min_version = MediaWikiVersion('1.27.0-wmf.13') - if MediaWikiVersion(self.version()) < min_version: + if self.mw_version < '1.27.0-wmf.13': raise FatalServerError(str(self) + ' version must be ' '1.27.0-wmf.13 or newer to support the ' 'history merge API.') @@ -5811,8 +5820,7 @@ revision = revision or set() # TODO: remove exeception for mw < 1.22 - if (revid or revision) and MediaWikiVersion( - self.version()) < MediaWikiVersion("1.22"): + if (revid or revision) and self.mw_version < '1.22': raise NotImplementedError( u'Support of "revid" parameter\n' u'is not implemented in MediaWiki version < "1.22"') @@ -5940,7 +5948,7 @@ 'token': self.tokens['watch'], 'unwatch': unwatch} unwatch = 'unwatched' if unwatch else 'watched' - if MediaWikiVersion(self.version()) >= MediaWikiVersion('1.23'): + if self.mw_version >= '1.23': parameters['titles'] = pages req = self._simple_request(**parameters) results = req.submit() @@ -6058,7 +6066,7 @@ upload action. """ - if MediaWikiVersion(self.version()) >= MediaWikiVersion('1.27wmf9'): + if self.mw_version >= '1.27wmf9': return not self._siteinfo.get('general')['uploadsenabled'] if hasattr(self, '_uploaddisabled'): return self._uploaddisabled @@ -6298,9 +6306,8 @@ # upload local file throttle = True filesize = os.path.getsize(source_filename) - chunked_upload = (chunk_size > 0 and chunk_size < filesize and - MediaWikiVersion( - self.version()) >= MediaWikiVersion('1.20')) + chunked_upload = ( + 0 < chunk_size < filesize and self.mw_version >= '1.20') with open(source_filename, 'rb') as f: final_request = self._request( throttle=throttle, parameters={ @@ -6824,7 +6831,7 @@ assert 'create' in self.protection_types(), \ "'create' should be a valid protection type." if type == 'create': - if MediaWikiVersion(self.version()) < MediaWikiVersion('1.15'): + if self.mw_version < '1.15': raise NotImplementedError( 'protectedpages(type=create) requires MW 1.15+') diff --git a/tests/api_tests.py b/tests/api_tests.py index 7e0d9b6..c31d5fd 100644 --- a/tests/api_tests.py +++ b/tests/api_tests.py @@ -20,7 +20,6 @@ from pywikibot.throttle import Throttle from pywikibot.tools import ( suppress_warnings, - MediaWikiVersion, PY2, UnicodeType, ) @@ -195,7 +194,7 @@ self.assertIn('main', pi._paraminfo) self.assertIn('paraminfo', pi._paraminfo) - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion("1.12"): + if self.site.mw_version >= '1.12': self.assertEqual(len(pi), len(pi.preloaded_modules)) @@ -235,7 +234,7 @@ self.assertIn('paraminfo', pi._paraminfo) self.assertIn('pageset', pi._paraminfo) - if MediaWikiVersion(self.site.version()) < MediaWikiVersion("1.12"): + if self.site.mw_version < '1.12': return if 'query' in pi.preloaded_modules: @@ -248,7 +247,7 @@ self.assertEqual(len(pi), len(pi.preloaded_modules)) - if MediaWikiVersion(site.version()) >= MediaWikiVersion("1.21"): + if site.mw_version >= '1.21': # 'generator' was added to 'pageset' in 1.21 generators_param = pi.parameter('pageset', 'generator') self.assertGreater(len(generators_param['type']), 1) @@ -265,7 +264,7 @@ self.assertIn('pageset', pi._paraminfo) self.assertIn('query', pi._paraminfo) - if MediaWikiVersion(site.version()) >= MediaWikiVersion("1.21"): + if site.mw_version >= '1.21': # 'generator' was added to 'pageset' in 1.21 pageset_generators_param = pi.parameter('pageset', 'generator') query_generators_param = pi.parameter('query', 'generator') @@ -282,7 +281,7 @@ self.assertIn('main', pi._paraminfo) self.assertIn('paraminfo', pi._paraminfo) - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion("1.12"): + if self.site.mw_version >= '1.12': self.assertEqual(len(pi), 1 + len(pi.preloaded_modules)) @@ -296,7 +295,7 @@ self.assertIsInstance(param['type'], list) - if MediaWikiVersion(self.site.version()) < MediaWikiVersion("1.12"): + if self.site.mw_version < '1.12': return self.assertIn('protection', param['type']) @@ -311,7 +310,7 @@ self.assertIn('main', pi._paraminfo) self.assertIn('paraminfo', pi._paraminfo) - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion("1.12"): + if self.site.mw_version >= '1.12': self.assertEqual(len(pi), 1 + len(pi.preloaded_modules)) @@ -325,7 +324,7 @@ self.assertIsInstance(param['type'], list) - if MediaWikiVersion(self.site.version()) < MediaWikiVersion("1.12"): + if self.site.mw_version < '1.12': return self.assertIn('user', param['type']) @@ -342,7 +341,7 @@ self.assertIn('main', pi._paraminfo) self.assertIn('paraminfo', pi._paraminfo) - if MediaWikiVersion(self.site.version()) < MediaWikiVersion("1.12"): + if self.site.mw_version < '1.12': return self.assertEqual(len(pi), @@ -368,7 +367,7 @@ self.assertIn('main', pi._paraminfo) self.assertIn('paraminfo', pi._paraminfo) - if MediaWikiVersion(self.site.version()) < MediaWikiVersion("1.12"): + if self.site.mw_version < '1.12': return self.assertEqual(len(pi), @@ -473,7 +472,7 @@ self.assertIn('main', pi._paraminfo) self.assertIn('paraminfo', pi._paraminfo) - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion("1.12"): + if self.site.mw_version >= '1.12': self.assertEqual(len(pi), 1 + len(pi.preloaded_modules)) @@ -482,10 +481,10 @@ def test_new_mode(self): """Test the new modules-only mode explicitly.""" site = self.get_site() - if MediaWikiVersion(site.version()) < MediaWikiVersion('1.25wmf4'): + if site.mw_version < '1.25wmf4': raise unittest.SkipTest( "version %s doesn't support the new paraminfo api" - % site.version()) + % site.mw_version) pi = api.ParamInfo(site, modules_only_mode=True) pi.fetch(['info']) self.assertIn('query+info', pi._paraminfo) @@ -786,7 +785,7 @@ params = { 'rvprop': 'ids|flags|timestamp|user|comment|content', 'titles': '|'.join(titles)} - if self.site.version() >= MediaWikiVersion('1.32'): + if self.site.mw_version >= '1.32': params['rvslots'] = 'main' gen = api.PropertyGenerator( site=self.site, diff --git a/tests/edit_tests.py b/tests/edit_tests.py index 62f8750..0c7206f 100644 --- a/tests/edit_tests.py +++ b/tests/edit_tests.py @@ -13,7 +13,6 @@ from pywikibot import config from pywikibot import page_put_queue -from pywikibot.tools import MediaWikiVersion from tests.aspects import unittest, TestCase from tests.oauth_tests import OAuthSiteTestCase @@ -88,7 +87,7 @@ dest = pywikibot.Page(site, 'User:Sn1per/MergeTest2') # Make sure the wiki supports action=mergehistory - if MediaWikiVersion(site.version()) < MediaWikiVersion('1.27.0-wmf.13'): + if site.mw_version < '1.27.0-wmf.13': raise unittest.SkipTest('Wiki version must be 1.27.0-wmf.13 or ' 'newer to support the history merge API.') diff --git a/tests/logentry_tests.py b/tests/logentry_tests.py index deb98f7..559cb2f 100644 --- a/tests/logentry_tests.py +++ b/tests/logentry_tests.py @@ -15,7 +15,6 @@ from pywikibot.logentries import ( LogEntryFactory, OtherLogEntry, UserTargetLogEntry) from pywikibot.tools import ( - MediaWikiVersion, UnicodeType as unicode, ) @@ -61,8 +60,7 @@ # This is an assertion as the tests don't make sense with newer # MW versions and otherwise it might not be visible that the test # isn't run on an older wiki. - self.assertLess(MediaWikiVersion(self.site.version()), - MediaWikiVersion('1.20')) + self.assertLess(self.site.mw_version, '1.20') return next(iter(self.site.logevents(logtype=logtype, total=1))) def _test_logevent(self, logtype): diff --git a/tests/page_tests.py b/tests/page_tests.py index f9d6601..bed1cc8 100644 --- a/tests/page_tests.py +++ b/tests/page_tests.py @@ -18,7 +18,6 @@ from pywikibot import InvalidTitle from pywikibot.tools import ( - MediaWikiVersion, PY2, StringTypes as basestring, suppress_warnings, @@ -579,7 +578,7 @@ mainpage = self.get_mainpage() image = pywikibot.FilePage(site, 'File:Jean-Léon Gérôme 003.jpg') - if MediaWikiVersion(site.version()) < MediaWikiVersion('1.20'): + if site.mw_version < '1.20': self.assertRaises(NotImplementedError, mainpage.page_image) elif site.has_extension('PageImages'): mainpage_image = mainpage.page_image() @@ -611,7 +610,7 @@ self.assertDeprecation() self._reset_messages() - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion('1.16'): + if self.site.mw_version >= '1.16': self.assertIsInstance(mainpage.previous_revision_id, int) self.assertEqual(mainpage.previous_revision_id, mainpage.latest_revision.parent_id) diff --git a/tests/paraminfo_tests.py b/tests/paraminfo_tests.py index aebc64e..4c6f757 100644 --- a/tests/paraminfo_tests.py +++ b/tests/paraminfo_tests.py @@ -11,7 +11,6 @@ from pywikibot.family import WikimediaFamily from pywikibot.page import Claim, Property from pywikibot.site import DataSite -from pywikibot.tools import MediaWikiVersion from tests.aspects import ( unittest, @@ -82,7 +81,7 @@ def test_watchlist_show_flags(self): """Test watchlist show flags.""" types = ['minor', 'bot', 'anon', 'patrolled'] - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion('1.24'): + if self.site.mw_version >= '1.24': types.append('unread') known = types + ['!%s' % item for item in types] @@ -93,12 +92,12 @@ """Test watchlist type.""" known = ['edit', 'new', 'log'] - _version = MediaWikiVersion(self.site.version()) + mw_ver = self.site.mw_version - if _version >= MediaWikiVersion('1.20'): + if mw_ver >= '1.20': known.append('external') - if _version.version >= (1, 27): - if _version >= MediaWikiVersion('1.27.0-wmf.4') or _version.suffix == 'alpha': + if mw_ver.version >= (1, 27): + if mw_ver >= '1.27.0-wmf.4' or mw_ver.suffix == 'alpha': known.append('categorize') self._check_param_values(self.site, 'query+watchlist', 'type', known) @@ -125,7 +124,7 @@ 'text/css', 'text/plain', ] - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion('1.24'): + if self.site.mw_version >= '1.24': base.append('application/json') if isinstance(self.site, DataSite): # It is not clear when this format has been added, see T129281. @@ -151,7 +150,7 @@ 'Scribunto', 'JsonSchema', ] - if MediaWikiVersion(self.site.version()) >= MediaWikiVersion('1.24'): + if self.site.mw_version >= '1.24': base.append('json') self._check_param_subset(self.site, 'edit', 'contentmodel', base) diff --git a/tests/site_tests.py b/tests/site_tests.py index cf7a9b4..ee7b90e 100644 --- a/tests/site_tests.py +++ b/tests/site_tests.py @@ -114,7 +114,7 @@ def test_siteinfo_normal_call(self): """Test calling the Siteinfo without setting dump.""" - if MediaWikiVersion(self.site.version()) < MediaWikiVersion('1.16'): + if self.site.mw_version < '1.16': raise unittest.SkipTest('requires v1.16+') old = self.site.siteinfo('general') @@ -317,7 +317,7 @@ # first-letter (== only first non-namespace letter is case insensitive) # See also: https://www.mediawiki.org/wiki/Manual:$wgCapitalLinks self.assertTrue(mysite.sametitle("Special:Always", "Special:always")) - if MediaWikiVersion(mysite.version()) >= MediaWikiVersion('1.16'): + if mysite.mw_version >= '1.16': self.assertTrue(mysite.sametitle('User:Always', 'User:always')) self.assertTrue(mysite.sametitle('MediaWiki:Always', 'MediaWiki:always')) @@ -1031,7 +1031,7 @@ def test_protectedpages_create(self): """Test that protectedpages returns protected page titles.""" - if MediaWikiVersion(self.site.version()) < MediaWikiVersion('1.15'): + if self.site.mw_version < '1.15': raise unittest.SkipTest('requires v1.15+') pages = list(self.get_site().protectedpages(type='create', total=10)) @@ -1075,7 +1075,7 @@ def test_pages_with_property(self): """Test pages_with_property method.""" - if MediaWikiVersion(self.site.version()) < MediaWikiVersion('1.21'): + if self.site.mw_version < '1.21': raise unittest.SkipTest('requires v1.21+') mysite = self.get_site() pnames = mysite.get_property_names() @@ -1440,7 +1440,7 @@ mysite = self.site mainpage = self.get_mainpage() imagepage = self.imagepage - if MediaWikiVersion(mysite.version()) <= MediaWikiVersion("1.14"): + if mysite.mw_version <= '1.14': pagelist = [mainpage] if imagepage: pagelist += [imagepage] @@ -2067,7 +2067,7 @@ self.assertIsInstance(result, dict) params = {'rcid': 0} - if mysite.version() >= MediaWikiVersion('1.22'): + if mysite.mw_version >= '1.22': params['revid'] = [0, 1] try: @@ -2149,7 +2149,7 @@ """Store version.""" super(TestSiteTokens, self).setUp() self.mysite = self.get_site() - self._version = MediaWikiVersion(self.mysite.version()) + self._version = self.mysite.mw_version self.orig_version = self.mysite.version def tearDown(self): @@ -2360,7 +2360,7 @@ def test_siteinfo_v1_16(self): """Test v.16+ siteinfo values.""" - if MediaWikiVersion(self.site.version()) < MediaWikiVersion('1.16'): + if self.site.mw_version < '1.16': raise unittest.SkipTest('requires v1.16+') mysite = self.get_site() diff --git a/tests/wikibase_edit_tests.py b/tests/wikibase_edit_tests.py index d5d8a02..63dee40 100644 --- a/tests/wikibase_edit_tests.py +++ b/tests/wikibase_edit_tests.py @@ -15,7 +15,6 @@ import time import pywikibot -from pywikibot.tools import MediaWikiVersion from tests.aspects import unittest, WikibaseTestCase @@ -242,8 +241,7 @@ item = self._clean_item(testsite, 'P20480') # Make sure the wiki supports wikibase-conceptbaseuri - version = testsite.version() - if MediaWikiVersion(version) < MediaWikiVersion('1.29.0-wmf.2'): + if testsite.mw_version < '1.29.0-wmf.2': raise unittest.SkipTest('Wiki version must be 1.29.0-wmf.2 or ' 'newer to support unbound uncertainties.') @@ -267,8 +265,7 @@ item = self._clean_item(testsite, 'P69') # Make sure the wiki supports unbound uncertainties - version = testsite.version() - if MediaWikiVersion(version) < MediaWikiVersion('1.29.0-wmf.2'): + if testsite.mw_version < '1.29.0-wmf.2': raise unittest.SkipTest('Wiki version must be 1.29.0-wmf.2 or ' 'newer to support unbound uncertainties.') @@ -290,8 +287,7 @@ item = self._clean_item(testsite, 'P69') # Make sure the wiki supports wikibase-conceptbaseuri - version = testsite.version() - if MediaWikiVersion(version) < MediaWikiVersion('1.28-wmf.23'): + if testsite.mw_version < '1.28-wmf.23': raise unittest.SkipTest('Wiki version must be 1.28-wmf.23 or ' 'newer to expose wikibase-conceptbaseuri.') diff --git a/tests/wikibase_tests.py b/tests/wikibase_tests.py index 7407847..81164d1 100644 --- a/tests/wikibase_tests.py +++ b/tests/wikibase_tests.py @@ -511,7 +511,7 @@ """Override setup to store repo and it's version.""" super(WikidataTestCase, self).setUp() self.repo = self.get_repo() - self.version = MediaWikiVersion(self.repo.version()) + self.version = self.repo.mw_version def test_WbQuantity_unbound(self): """Test WbQuantity for value without bounds.""" -- To view, visit https://gerrit.wikimedia.org/r/455505 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I75de443b24d2eb478724c416db8f2d734b6bb806 Gerrit-Change-Number: 455505 Gerrit-PatchSet: 5 Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75) Gerrit-CC: Zhuyifei1999 <zhuyifei1999(a)gmail.com>

1 0

[Gerrit] pywikibot/core[master]: [cleanup] move code parts from create_user_config to copy_sections
by jenkins-bot (Code Review) 27 Aug '18

27 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/455179 ) Change subject: [cleanup] move code parts from create_user_config to copy_sections ...................................................................... [cleanup] move code parts from create_user_config to copy_sections Partly detached from I68a04e9198dc179f06c6245f1c1aec518603bd34 - There is no warning anymore if no section or only few lines where found. This part has been moved to test suite. Change-Id: I86f06d4ec7e3a93cc217145ed659be2e8dc64d3b --- M generate_user_files.py M tests/generate_user_files_tests.py 2 files changed, 73 insertions(+), 43 deletions(-) Approvals: Framawiki: Looks good to me, approved jenkins-bot: Verified diff --git a/generate_user_files.py b/generate_user_files.py index 57dda8d..bef3b80 100755 --- a/generate_user_files.py +++ b/generate_user_files.py @@ -15,7 +15,6 @@ import sys from textwrap import fill -from warnings import warn from generate_family_file import _import_with_no_user_config @@ -231,6 +230,42 @@ {botpasswords}""" +def copy_sections(): + """Take config sections and copying them to user-config.py. + + config2.py will be in the pywikibot/ directory relative to this + generate_user_files script. + + @return: config text of all sections. + @rtype: str + """ + install = os.path.dirname(os.path.abspath(__file__)) + with codecs.open(os.path.join(install, 'pywikibot', 'config2.py'), + 'r', 'utf-8') as config_f: + config_file = config_f.read() + + result = re.findall( + '^(# ############# (?:' + 'LOGFILE|' + 'EXTERNAL SCRIPT PATH|' + 'INTERWIKI|' + 'SOLVE_DISAMBIGUATION|' + 'IMAGE RELATED|' + 'TABLE CONVERSION BOT|' + 'WEBLINK CHECKER|' + 'DATABASE|' + 'SEARCH ENGINE|' + 'COPYRIGHT|' + 'FURTHER' + ') SETTINGS .*)^(?=#####|# =====)', + config_file, re.MULTILINE | re.DOTALL) + + if not result: # Something is wrong with the regex + return None + + return '\n'.join(result) + + def create_user_config(main_family, main_code, main_username, force=False): """ Create a user-config.py in base_dir. @@ -292,49 +327,12 @@ "('{0}', BotPassword('{1}', '{2}'))".format(*botpassword) for botpassword in botpasswords) - config_text = '' - config_content = SMALL_CONFIG - - try: - # config2.py will be in the pywikibot/ directory relative to this - # script (generate_user_files) - install = os.path.dirname(os.path.abspath(__file__)) - with codecs.open(os.path.join(install, "pywikibot", "config2.py"), - "r", "utf-8") as config_f: - config_file = config_f.read() - - res = re.findall("^(# ############# (?:" - "LOGFILE|" - 'EXTERNAL SCRIPT PATH|' - "INTERWIKI|" - "SOLVE_DISAMBIGUATION|" - "IMAGE RELATED|" - "TABLE CONVERSION BOT|" - "WEBLINK CHECKER|" - "DATABASE|" - "SEARCH ENGINE|" - "COPYRIGHT|" - "FURTHER" - ") SETTINGS .*)^(?=#####|# =====)", - config_file, re.MULTILINE | re.DOTALL) - - if not res: - warn('Extended config extraction failed', UserWarning) - - config_text = '\n'.join(res) - if len(config_text.splitlines()) < 350: - warn('Extended config extraction too short: %d' - % len(config_text.splitlines()), - UserWarning) - + config_text = copy_sections() + if config_text: config_content = EXTENDED_CONFIG - except Exception as e: - # If the warning was explicitly enabled, raise - if isinstance(e, UserWarning): - raise - pywikibot.output('Exception while creating extended user-config; ' - 'falling back to simple user-config.') - pywikibot.exception() + else: + pywikibot.output('Creating a small variant of user-config.py') + config_content = SMALL_CONFIG try: # Finally save user-config.py diff --git a/tests/generate_user_files_tests.py b/tests/generate_user_files_tests.py index 743543a..bde7497 100644 --- a/tests/generate_user_files_tests.py +++ b/tests/generate_user_files_tests.py @@ -10,6 +10,7 @@ import re from tests.aspects import unittest, TestCase +from unittest import expectedFailure import generate_user_files as guf @@ -67,6 +68,37 @@ self.assertEqual(code, 'test') self.assertEqual(user, 'bar') + @expectedFailure # T145371 + def test_copy_sections_fail(self): + """Test copy_sections function for sections not in config text.""" + config_text = guf.copy_sections() + for section in ('HTTP SETTINGS', + 'REPLICATION BOT SETTINGS', + ): + self.assertNotIn(section, config_text) + + def test_copy_sections_not_found(self): + """Test copy_sections function for sections not in config text.""" + config_text = guf.copy_sections() + for section in ('ACCOUNT SETTINGS', + 'OBSOLETE SETTINGS', + 'EXTERNAL EDITOR SETTINGS', + ): + self.assertNotIn(section, config_text) + + def test_copy_sections_found(self): + """Test copy_sections function for sections found in config text.""" + config_text = guf.copy_sections() + self.assertIsNotNone(config_text) + for section in ('LOGFILE SETTINGS', + 'EXTERNAL SCRIPT PATH SETTINGS', + 'INTERWIKI SETTINGS', + 'FURTHER SETTINGS', + ): + self.assertIn(section, config_text) + lines = config_text.splitlines() + self.assertGreater(len(lines), 350) + if __name__ == '__main__': # pragma: no cover try: -- To view, visit https://gerrit.wikimedia.org/r/455179 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I86f06d4ec7e3a93cc217145ed659be2e8dc64d3b Gerrit-Change-Number: 455179 Gerrit-PatchSet: 5 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

1 0

[Gerrit] pywikibot/core[master]: [IMPR] improve hash method for BasePage and Link
by jenkins-bot (Code Review) 24 Aug '18

24 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/453576 ) Change subject: [IMPR] improve hash method for BasePage and Link ...................................................................... [IMPR] improve hash method for BasePage and Link - use _cmpkey to build hash BasePage - simplify Link's hash method by using sitename and canonical_title as tuple Change-Id: I346b050665b3337cca0ff9e22e8d493547b2a7af --- M pywikibot/page.py 1 file changed, 2 insertions(+), 4 deletions(-) Approvals: Framawiki: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index e2f2fcc..46021a9 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -413,7 +413,7 @@ This relies on the fact that the string representation of an instance can not change after the construction. """ - return hash(unicode(self)) + return hash(self._cmpkey()) def full_url(self): """Return the full URL.""" @@ -5846,9 +5846,7 @@ def __hash__(self): """A stable identifier to be used as a key in hash-tables.""" - return hash(u'%s:%s:%s' % (self.site.family.name, - self.site.code, - self.title)) + return hash((self.site.sitename, self.canonical_title())) @classmethod def fromPage(cls, page, source=None): -- To view, visit https://gerrit.wikimedia.org/r/453576 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I346b050665b3337cca0ff9e22e8d493547b2a7af Gerrit-Change-Number: 453576 Gerrit-PatchSet: 4 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75) Gerrit-CC: Matěj Suchánek <matejsuchanek97(a)gmail.com>

1 0

[Gerrit] pywikibot/core[master]: [cleanup] cleanup config2.py
by jenkins-bot (Code Review) 24 Aug '18

24 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/455139 ) Change subject: [cleanup] cleanup config2.py ...................................................................... [cleanup] cleanup config2.py - fix section header for "EXTERNAL SCRIPT PATH SETTINGS" and "REPLICATION BOT SETTINGS" - separate section info text into a header line and additional comments by a new line detached from I68a04e9198dc179f06c6245f1c1aec518603bd34 Change-Id: Ic04c36407f0a0b9f58791e93c8519c54ad2618df --- M pywikibot/config2.py 1 file changed, 16 insertions(+), 9 deletions(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/config2.py b/pywikibot/config2.py index 552f060..ad97656 100644 --- a/pywikibot/config2.py +++ b/pywikibot/config2.py @@ -519,6 +519,7 @@ # Defines for which scripts a logfile should be enabled. Logfiles will be # saved in the 'logs' subdirectory. +# # Example: # log = ['interwiki', 'weblinkchecker', 'table2wiki'] # It is also possible to enable logging for all scripts, using this line: @@ -550,9 +551,10 @@ # (overrides log setting above) debug_log = [] -# ############# EXTERNAL SCRIPT PATH SETTING ############## -# set your own script path to lookup for your script files. -# your private script path must be located inside the +# ############# EXTERNAL SCRIPT PATH SETTINGS ############## +# Set your own script path to lookup for your script files. +# +# Your private script path must be located inside the # framework folder, subfolders must be delimited by '.'. # every folder must contain an (empty) __init__.py file. # @@ -619,6 +621,7 @@ # # Set disambiguation_comment[FAMILY][LANG] to a non-empty string to override # the default edit comment for the solve_disambiguation bot. +# # Use %s to represent the name of the disambiguation page being treated. # Example: # @@ -639,6 +642,7 @@ # 'minthrottle' seconds. This can be lengthened if the server is slow, # but never more than 'maxthrottle' seconds. However - if you are running # more than one bot in parallel the times are lengthened. +# # By default, the get_throttle is turned off, and 'maxlag' is used to # control the rate of server access. Set minthrottle to non-zero to use a # throttle on read access. @@ -674,8 +678,8 @@ # ############# TABLE CONVERSION BOT SETTINGS ############## -# will split long paragraphs for better reading the source. -# only table2wiki.py use it by now +# Will split long paragraphs for better reading the source. +# Only table2wiki.py use it by now. splitLongParagraphs = False # sometimes HTML-tables are indented for better reading. # That can do very ugly results. @@ -696,6 +700,7 @@ # ############# DATABASE SETTINGS ############## # Setting to connect the database or replica of the database of the wiki. # db_name_format can be used to manipulate the dbName of site. +# # Example for a pywikibot running on wmflabs: # db_hostname = 'enwiki.analytics.db.svc.eqiad.wmflabs' # db_name_format = '{0}_p' @@ -711,7 +716,8 @@ db_port = 3306 # ############# SEARCH ENGINE SETTINGS ############## - +# Live search web service appid settings. +# # Yahoo! Search Web Services are not operational. # See https://phabricator.wikimedia.org/T106085 yahoo_appid = '' @@ -838,9 +844,10 @@ cosmetic_changes_deny_script = ['category_redirect', 'cosmetic_changes', 'newitem', 'touch'] -# ############# REPLICATION BOT ################ -# You can add replicate_replace to your user-config.py, which has the following -# format: +# ############# REPLICATION BOT SETTINGS ################ +# You can add replicate_replace to your user-config.py. +# +# Use has the following format: # # replicate_replace = { # 'wikipedia:li': {'Hoofdpagina': 'Veurblaad'} -- To view, visit https://gerrit.wikimedia.org/r/455139 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic04c36407f0a0b9f58791e93c8519c54ad2618df Gerrit-Change-Number: 455139 Gerrit-PatchSet: 3 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

1 0

[Gerrit] pywikibot/core[master]: pagegenerators.py: Avoid applying two uniquifying filters
by jenkins-bot (Code Review) 24 Aug '18

24 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/454198 ) Change subject: pagegenerators.py: Avoid applying two uniquifying filters ...................................................................... pagegenerators.py: Avoid applying two uniquifying filters GeneratorFactory applies unique_filter at two levels under certain conditions, see T199615#4517616. This happens because UserContributionsGenerator and RecentChangesPageGenerator are always used with filter_unique and when combined, the combination is also passed through filter_unique. This might lead to unnecessary memory consumption. To resolve the issue, we will apply filter_unique only once, after all generators are created. - Create global function `_filter_unique_pages`. It will help in applying unique_filter to page generators without having to pass the `key` argument every time. - Remove `GeneratorFactory._filter_unique`. Use the global `_filter_unique_pages` Instead. - Do not pre-apply filter_unique to UserContributionsGenerator and RecentChangesPageGenerator. Instead use `self._single_gen_filter_unique` as a flag to indicate that the results should be filtered later. - Convert `GeneratorFactory._parse_log_events` into a static method. - getCombinedGenerator: Remove `gensList` variable. It was not used anywhere else. Bug: T199615 Change-Id: Ida8005c8fe6f48d287c106983350cb09b8722134 --- M pywikibot/pagegenerators.py 1 file changed, 25 insertions(+), 22 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index ec5a83b..7378a96 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -395,6 +395,11 @@ __doc__ = __doc__.replace("&params;", parameterHelp) +# This is the function that will be used to de-duplicate page iterators. +_filter_unique_pages = partial( + filter_unique, key=lambda page: '{}:{}:{}'.format(*page._cmpkey())) + + class GeneratorFactory(object): """Process command line arguments and return appropriate page generator. @@ -403,14 +408,6 @@ that are used by many scripts and that determine which pages to work on. """ - # This is the function that will be used to de-duplicate iterators. - _filter_unique = staticmethod(partial( - filter_unique, key=lambda p: '{}:{}:{}'.format(*p._cmpkey()))) - # The seen list can not yet be shared at present, due to `intersect` mode - # not being known until after all generators have been created. - # When not in intersect mode, _filter_unique could be: - # functools.partial(filter_unique, container=global_seen_list, key=...) - def __init__(self, site=None, positional_arg_name=None): """ Initializer. @@ -517,19 +514,19 @@ 'filter(s) specified but no generators.') return None elif len(self.gens) == 1: - gensList = self.gens[0] - dupfiltergen = gensList + dupfiltergen = self.gens[0] + if hasattr(self, '_single_gen_filter_unique'): + dupfiltergen = _filter_unique_pages(dupfiltergen) if self.intersect: pywikibot.warning( '"-intersect" ignored as only one generator is specified.') else: if self.intersect: - gensList = intersect_generators(self.gens) # By definition no duplicates are possible. - dupfiltergen = gensList + dupfiltergen = intersect_generators(self.gens) else: - gensList = itertools.chain(*self.gens) - dupfiltergen = self._filter_unique(gensList) + dupfiltergen = _filter_unique_pages(itertools.chain( + *self.gens)) # Add on subpage filter generator if self.subpage_max_depth is not None: @@ -627,7 +624,8 @@ recurse=recurse, content=content) - def _parse_log_events(self, logtype, user=None, start=None, end=None): + @staticmethod + def _parse_log_events(logtype, user=None, start=None, end=None): """ Parse the -logevent argument information. @@ -758,7 +756,9 @@ def _handle_usercontribs(self, value): """Handle `-usercontribs` argument.""" - return UserContributionsGenerator(value, site=self.site) + self._single_gen_filter_unique = True + return UserContributionsGenerator( + value, site=self.site, _filter_unique=None) def _handle_withoutinterwiki(self, value): """Handle `-withoutinterwiki` argument.""" @@ -815,9 +815,10 @@ rcend = rcstart - timedelta(minutes=duration) elif len(params) == 1: total = int(params[0]) + self._single_gen_filter_unique = True return RecentChangesPageGenerator( namespaces=self.namespaces, total=total, start=rcstart, end=rcend, - site=self.site, tag=rctag, _filter_unique=self._filter_unique) + site=self.site, tag=rctag) def _handle_liverecentchanges(self, value): """Handle `-liverecentchanges` argument.""" @@ -1609,7 +1610,8 @@ @deprecated_args(number='total', step=None) def UserContributionsGenerator(username, namespaces=None, site=None, - total=None, _filter_unique=filter_unique): + total=None, + _filter_unique=_filter_unique_pages): """Yield unique pages edited by user:username. @param total: Maximum number of pages to retrieve in total @@ -1627,10 +1629,11 @@ pywikibot.warning('User "{}" does not exist on site "{}".' .format(user.username, site)) - return _filter_unique( - (contrib[0] for contrib in user.contributions( - namespaces=namespaces, total=total)), - key=lambda p: '{}:{}:{}'.format(*p._cmpkey())) + gen = (contrib[0] for contrib in user.contributions( + namespaces=namespaces, total=total)) + if _filter_unique: + return _filter_unique(gen) + return gen def NamespaceFilterPageGenerator(generator, namespaces, site=None): -- To view, visit https://gerrit.wikimedia.org/r/454198 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ida8005c8fe6f48d287c106983350cb09b8722134 Gerrit-Change-Number: 454198 Gerrit-PatchSet: 5 Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

1 0

[Gerrit] pywikibot/core[master]: CosmeticChangesToolkit.removeEmptySections: Fix skipping of language ...
by jenkins-bot (Code Review) 24 Aug '18

24 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/454853 ) Change subject: CosmeticChangesToolkit.removeEmptySections: Fix skipping of language links ...................................................................... CosmeticChangesToolkit.removeEmptySections: Fix skipping of language links The 'interwiki' regex includes the trailing newlines and therefore may potentially mess up page sections. (T202629) Instead of using the 'interwiki' regex, use the textlib.removeLanguageLinks function. Use marker='\n' argument to avoid the issue mentioned above. Add a few tests for the fixed bug. Do minor cleanups in textlib.py: - _ignore_case: No need to ignore-case a letter if it is not case-sensitive. - 'interwiki' regex patter, replace '[\s]*' with '\s*'. Bug: T202629 Change-Id: I22b4ad92c7058ca65c6e3e6a680586dc1c2490a0 --- M pywikibot/cosmetic_changes.py M pywikibot/textlib.py M tests/cosmetic_changes_tests.py 3 files changed, 20 insertions(+), 8 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/cosmetic_changes.py b/pywikibot/cosmetic_changes.py index 3130081..ac4d548 100755 --- a/pywikibot/cosmetic_changes.py +++ b/pywikibot/cosmetic_changes.py @@ -663,12 +663,8 @@ def removeEmptySections(self, text): """Cleanup empty sections.""" - # comments, categories, and interwikis - skippings = ['comment', 'category', 'interwiki'] + skippings = ['comment', 'category'] skip_regexes = _get_regexes(skippings, self.site) - # we want only interwikis, not interlanguage links - skip_regexes[1] = re.compile( - skip_regexes[1].pattern.replace(':?', '')) # site defined templates skip_templates = { 'cs': ('Pahýl[ _]část',), # stub section @@ -681,7 +677,7 @@ skip_regexes.append(re.compile(r'(?m)^[\*#] *$')) # get stripped sections - stripped_text = text + stripped_text = textlib.removeLanguageLinks(text, self.site, '\n') for reg in skip_regexes: stripped_text = reg.sub(r'', stripped_text) strip_sections = textlib.extract_sections( diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py index 074ba72..c9d416b 100644 --- a/pywikibot/textlib.py +++ b/pywikibot/textlib.py @@ -251,7 +251,9 @@ def _ignore_case(string): """Return a case-insensitive pattern for the string.""" - return ''.join('[' + c.upper() + c.lower() + ']' for c in string) + return ''.join( + '[' + c + s + ']' if c != s else c + for s, c in zip(string, string.swapcase())) def _tag_pattern(tag_name): @@ -282,7 +284,7 @@ 'hyperlink': compileLinkR(), # also finds links to foreign sites with preleading ":" 'interwiki': ( - r'\[\[:?(%s)\s?:[^\]]*\]\][\s]*', + r'\[\[:?(%s)\s?:[^\]]*\]\]\s*', lambda site: '|'.join( _ignore_case(i) for i in site.validLanguageLinks() + list(site.family.obsolete.keys()))), diff --git a/tests/cosmetic_changes_tests.py b/tests/cosmetic_changes_tests.py index d5354c0..a6024a2 100644 --- a/tests/cosmetic_changes_tests.py +++ b/tests/cosmetic_changes_tests.py @@ -332,6 +332,20 @@ self.cct.removeEmptySections('\n==Bar==\n[[cs:Foo]]' '\n[[Category:Baz]]')) + def test_remove_empty_sections_interlanguage_links(self): + """Test removeEmptySections with edge cases of language links.""" + # When removing language links, do not remove the \n after them, + # otherwise the sections won't be detected correctly. + text = 'text [[:en:link]]\n=== title1 ===\ncontent1' + self.assertEqual(text, self.cct.removeEmptySections(text)) + self.assertEqual( + 't [[en:link]]\n=== 1 ===\nc', + self.cct.removeEmptySections('t [[en:link]]\n=== 1 ===\nc')) + # Treat sections that only contain language links as empty sections. + self.assertEqual( + 't\n[[en:link]]', + self.cct.removeEmptySections('t\n=== 1 ===\n[[en:link]]')) + def test_remove_empty_sections_with_heading_comments(self): """Test removeEmptySections with comments in the section headings.""" self.assertEqual( -- To view, visit https://gerrit.wikimedia.org/r/454853 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I22b4ad92c7058ca65c6e3e6a680586dc1c2490a0 Gerrit-Change-Number: 454853 Gerrit-PatchSet: 9 Gerrit-Owner: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

1 0

[Gerrit] pywikibot/core[master]: [doc] Prepare next release
by jenkins-bot (Code Review) 23 Aug '18

23 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/454861 ) Change subject: [doc] Prepare next release ...................................................................... [doc] Prepare next release Change-Id: I617dc550b618a2d95116785cf864bf4cb34110d1 --- M HISTORY.rst M docs/conf.py 2 files changed, 7 insertions(+), 1 deletion(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/HISTORY.rst b/HISTORY.rst index b2e5ed2..ca50425 100644 --- a/HISTORY.rst +++ b/HISTORY.rst @@ -4,6 +4,12 @@ Current release --------------- +* Bugfixes and improvements +* Localisation updates + +3.0.20180823 +------------ + * Don't reset Bot._site to None if we have already a site object (T125046) * pywikibot.site.Siteinfo: Fix the bug in cache_time when loading a CachedRequest (T202227) * pagegenerators._handle_recentchanges: Do not request for reversed results (T199199) diff --git a/docs/conf.py b/docs/conf.py index 3cf3d95..e08c3d4 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -71,7 +71,7 @@ # The short X.Y version. version = '3.0' # The full version, including alpha/beta/rc tags. -release = '3.0.20180710' +release = '3.0.20180823' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. -- To view, visit https://gerrit.wikimedia.org/r/454861 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I617dc550b618a2d95116785cf864bf4cb34110d1 Gerrit-Change-Number: 454861 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

1 0

[Gerrit] pywikibot/core[master]: [doc] update HISTORY.rst
by jenkins-bot (Code Review) 23 Aug '18

23 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/454793 ) Change subject: [doc] update HISTORY.rst ...................................................................... [doc] update HISTORY.rst Change-Id: I30e36f06887eb5b4e565410298cf8952a61adc76 --- M HISTORY.rst 1 file changed, 1 insertion(+), 0 deletions(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/HISTORY.rst b/HISTORY.rst index f856aad..b2e5ed2 100644 --- a/HISTORY.rst +++ b/HISTORY.rst @@ -4,6 +4,7 @@ Current release --------------- +* Don't reset Bot._site to None if we have already a site object (T125046) * pywikibot.site.Siteinfo: Fix the bug in cache_time when loading a CachedRequest (T202227) * pagegenerators._handle_recentchanges: Do not request for reversed results (T199199) * Use a key for filter_unique where appropriate (T199615) -- To view, visit https://gerrit.wikimedia.org/r/454793 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I30e36f06887eb5b4e565410298cf8952a61adc76 Gerrit-Change-Number: 454793 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

1 0

[Gerrit] pywikibot/core[master]: [bugfix] Don't reset Bot._site to None if we have already a site object
by jenkins-bot (Code Review) 23 Aug '18

23 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/266994 ) Change subject: [bugfix] Don't reset Bot._site to None if we have already a site object ...................................................................... [bugfix] Don't reset Bot._site to None if we have already a site object Bug: T125046 Change-Id: Id900d93b6235ea4ec60a0db97d7b7da2828fcaad --- M pywikibot/bot.py 1 file changed, 6 insertions(+), 2 deletions(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/bot.py b/pywikibot/bot.py index 2708755..d450a6d 100644 --- a/pywikibot/bot.py +++ b/pywikibot/bot.py @@ -1537,11 +1537,15 @@ instead which specifically handle multiple or single sites. """ - def __init__(self, **kwargs): + def __init__(self, site=None, **kwargs): """Create a Bot instance and initalize cached sites.""" # TODO: add warning if site is specified and generator # contains pages from a different site. - self._site = kwargs.pop('site', None) + # Do not set self._site to None if we already have it + if site is not None: + self._site = site + elif not hasattr(self, '_site'): + self._site = None self._sites = set([self._site] if self._site else []) super(Bot, self).__init__(**kwargs) -- To view, visit https://gerrit.wikimedia.org/r/266994 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id900d93b6235ea4ec60a0db97d7b7da2828fcaad Gerrit-Change-Number: 266994 Gerrit-PatchSet: 3 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot (75)

1 0

[Gerrit] pywikibot/core[master]: [doc] Update HISTORY.rst
by jenkins-bot (Code Review) 23 Aug '18

23 Aug '18

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/454777 ) Change subject: [doc] Update HISTORY.rst ...................................................................... [doc] Update HISTORY.rst Change-Id: I2a30d575a0cf6d12c32d73a66c83293a09932eb6 --- M HISTORY.rst 1 file changed, 17 insertions(+), 0 deletions(-) Approvals: Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/HISTORY.rst b/HISTORY.rst index 09bd15b..f856aad 100644 --- a/HISTORY.rst +++ b/HISTORY.rst @@ -4,7 +4,24 @@ Current release --------------- +* pywikibot.site.Siteinfo: Fix the bug in cache_time when loading a CachedRequest (T202227) +* pagegenerators._handle_recentchanges: Do not request for reversed results (T199199) +* Use a key for filter_unique where appropriate (T199615) +* pywikibot.tools: Add exceptions for first_upper (T200357) +* Fix usages of site.namespaces.NAMESPACE_NAME (T201969) +* pywikibot/textlib.py: Fix header regex to allow comments +* Use 'rvslots' when fetching revisions on MW 1.32+ (T200955) * Drop the '2' from PYWIKIBOT2_DIR, PYWIKIBOT2_DIR_PWB, and PYWIKIBOT2_NO_USER_CONFIG environment variables. The old names are now deprecated. The other PYWIKIBOT2_* variables which were used only for testing purposes have been renamed without deprecation. (T184674) +* Introduce a timestamp in deprecated decorator (T106121) +* textlib.extract_sections: Remove footer from the last section (T199751) +* Don't let WikidataBot crash on save related errors (T199642) +* Allow different projects to have different L10N entries (T198889) +* remove color highlights before fill function (T196874) +* Fix Portuguese file namespace translation in cc (T57242) +* textlib._create_default_regexes: Avoid using inline flags (T195538) +* Not everything after a language link is footer (T199539) +* code cleanups +* New mediawiki projects were provided * Bugfixes and improvements * Localisation updates -- To view, visit https://gerrit.wikimedia.org/r/454777 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2a30d575a0cf6d12c32d73a66c83293a09932eb6 Gerrit-Change-Number: 454777 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Zoranzoki21 <zorandori4444(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits August 2018