Pywikibot-commits November 2014

pywikibot-commits@lists.wikimedia.org

2 participants
170 discussions

[Gerrit] [FIX] Upload: Use fake filename in MIME header - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [FIX] Upload: Use fake filename in MIME header ...................................................................... [FIX] Upload: Use fake filename in MIME header The standard MIME request does only support ASCII characters in the headers. As the value is apparently not interpreted it's replaced by a dummy value, so Unicode characters in the file page name doesn't cause pywikibot to receive an API error. There is RFC2231 which is supported by both Python versions but not by PHP on which the MediaWiki server rely (see bug: 73662). This does only fix the uploads as it is unknown how to handle other MIME requests. Bug: 73661 Change-Id: I5ea92dcf1115b675b56142494d805e75d389b7d4 --- M pywikibot/site.py 1 file changed, 4 insertions(+), 2 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/site.py b/pywikibot/site.py index 5b1399d..0093c79 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -4460,6 +4460,8 @@ result = None file_page_title = filepage.title(withNamespace=False) if source_filename: + # TODO: Dummy value to allow also Unicode names, see bug 73661 + mime_filename = 'FAKE-NAME' # upload local file # make sure file actually exists if not os.path.isfile(source_filename): @@ -4483,7 +4485,7 @@ mime_params={}, throttle=throttle) req.mime_params['chunk'] = (chunk, ("application", "octet-stream"), - {'filename': file_page_title}) + {'filename': mime_filename}) if file_key: req['filekey'] = file_key try: @@ -4518,7 +4520,7 @@ 'mime_params': { 'file': (file_contents, filetype.split('/'), - {'filename': file_page_title}) + {'filename': mime_filename}) } } req = api.Request(site=self, action="upload", token=token, -- To view, visit https://gerrit.wikimedia.org/r/174677 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5ea92dcf1115b675b56142494d805e75d389b7d4 Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Matanya <matanya(a)foss.co.il> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] Fix tests after improvements in APISite.sametitle - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Fix tests after improvements in APISite.sametitle ...................................................................... Fix tests after improvements in APISite.sametitle The preloading tests where written before APISite.sametitle was improved, however they landed after the sametitle change. As a result, most of the builds broke. Change-Id: I3381d693fb3d03708bcb0218d0ad95d90f1f8002 --- M tests/site_tests.py 1 file changed, 6 insertions(+), 4 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/tests/site_tests.py b/tests/site_tests.py index ca605d8..e738859 100644 --- a/tests/site_tests.py +++ b/tests/site_tests.py @@ -1685,6 +1685,7 @@ # preloadpages will send the page ids, as they have already been loaded # by pagelinks, and preloadpages should complain the returned titles # do not match any title in the pagelist. + # However, APISite.sametitle now correctly links them. for page in links: page._link._text += ' ' @@ -1693,8 +1694,8 @@ self.assertIsInstance(page, pywikibot.Page) self.assertIsInstance(page.exists(), bool) if page.exists(): - self.assertFalse(hasattr(page, "_text")) - self.assertEqual(len(page._revisions), 0) + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) self.assertFalse(hasattr(page, '_pageprops')) count += 1 if count > 5: @@ -1712,6 +1713,7 @@ # change the title of the page _and_ delete the pageids. # preloadpages can only send the titles, and preloadpages should # complain the returned titles do not match any title in the pagelist. + # However, APISite.sametitle now correctly links them. for page in links: page._link._text += ' ' del page._pageid @@ -1721,8 +1723,8 @@ self.assertIsInstance(page, pywikibot.Page) self.assertIsInstance(page.exists(), bool) if page.exists(): - self.assertFalse(hasattr(page, "_text")) - self.assertEqual(len(page._revisions), 0) + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) self.assertFalse(hasattr(page, '_pageprops')) count += 1 if count > 5: -- To view, visit https://gerrit.wikimedia.org/r/174753 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I3381d693fb3d03708bcb0218d0ad95d90f1f8002 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] Fix noreferences xml support - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Fix noreferences xml support ...................................................................... Fix noreferences xml support The initial port imported xmlreader instead of pywikibot.xmlreader. Change-Id: Ia92826d131307525cb599da4f835fc8a860492f9 --- M scripts/noreferences.py 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/noreferences.py b/scripts/noreferences.py index 3ca9711..d0e7ce0 100755 --- a/scripts/noreferences.py +++ b/scripts/noreferences.py @@ -42,7 +42,9 @@ # import re + import pywikibot + from pywikibot import i18n, pagegenerators, textlib, Bot # This is required for the text that is shown when you run this script @@ -446,7 +448,7 @@ self.referencesR = re.compile('<references.*?/>', re.IGNORECASE) def __iter__(self): - import xmlreader + from pywikibot import xmlreader dump = xmlreader.XmlDump(self.xmlFilename) for entry in dump.parse(): text = textlib.removeDisabledParts(entry.text) -- To view, visit https://gerrit.wikimedia.org/r/174605 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia92826d131307525cb599da4f835fc8a860492f9 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] Disable decorators and wrappers when not __debug__ - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Disable decorators and wrappers when not __debug__ ...................................................................... Disable decorators and wrappers when not __debug__ When python is called with -O, it optimises the code and sets __debug__ to False, which may be used to disable blocks of code not suitable for an optimised tool. This patch disables decorators and wrappers, which provide deprecation facilities or check the server version and user rights before performing an operation. In optimise mode the caller must explicitly login, avoid deprecated arguments, check the server versions & features. epydoc can be generated using: python -O /usr/bin/epydoc pywikibot Bug: 72424 Change-Id: I67ed6b3e79ac46ef92e2b08ef6c606fd7177bae2 --- M pywikibot/site.py M pywikibot/tools.py M tests/aspects.py M tests/deprecation_tests.py M tests/dry_site_tests.py 5 files changed, 46 insertions(+), 7 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve XZise: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/site.py b/pywikibot/site.py index c727248..dd994e2 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -963,6 +963,10 @@ else: raise Exception("Not implemented") return fn(self, *args, **kwargs) + + if not __debug__: + return fn + callee.__name__ = fn.__name__ callee.__doc__ = fn.__doc__ callee.__module__ = callee.__module__ @@ -990,6 +994,10 @@ u"isn't implemented in MediaWiki version < %s" % (fn.__name__, version)) return fn(self, *args, **kwargs) + + if not __debug__: + return fn + callee.__name__ = fn.__name__ callee.__doc__ = fn.__doc__ callee.__module__ = fn.__module__ diff --git a/pywikibot/tools.py b/pywikibot/tools.py index ad890eb..d71ad8a 100644 --- a/pywikibot/tools.py +++ b/pywikibot/tools.py @@ -438,6 +438,7 @@ """Python 3 iterator method.""" return self.next() + # Decorators # # Decorator functions without parameters are _invoked_ differently from @@ -536,6 +537,9 @@ else: return inner_wrapper + if not __debug__: + return obj + return outer_wrapper @@ -571,6 +575,9 @@ warning(u"%s is deprecated." % (name)) return obj(*args, **kwargs) + if not __debug__: + return obj + wrapper.__doc__ = obj.__doc__ wrapper.__name__ = obj.__name__ wrapper.__module__ = obj.__module__ @@ -586,6 +593,9 @@ # When called as @deprecated, return a replacement function if without_parameters: + if not __debug__: + return args[0] + return decorator(args[0]) # Otherwise return a decorator, which returns a replacement function else: @@ -646,6 +656,9 @@ del __kw[old_arg] return obj(*__args, **__kw) + if not __debug__: + return obj + wrapper.__doc__ = obj.__doc__ wrapper.__name__ = obj.__name__ wrapper.__module__ = obj.__module__ @@ -704,6 +717,10 @@ old=old_name or target.__name__, target=target_module, source=source_module) + + if not __debug__: + return target + return call @@ -726,7 +743,9 @@ super(ModuleDeprecationWrapper, self).__setattr__('_deprecated', {}) super(ModuleDeprecationWrapper, self).__setattr__('_module', module) super(ModuleDeprecationWrapper, self).__setattr__('__doc__', module.__doc__) - sys.modules[module.__name__] = self + + if __debug__: + sys.modules[module.__name__] = self def _add_deprecated_attr(self, name, replacement=None, replacement_name=None): diff --git a/tests/aspects.py b/tests/aspects.py index f134dcc..00ee838 100644 --- a/tests/aspects.py +++ b/tests/aspects.py @@ -915,7 +915,19 @@ user = True -class DeprecationTestCase(TestCase): +class DebugOnlyTestCase(TestCase): + + """Test cases that only operate in debug mode.""" + + @classmethod + def setUpClass(cls): + if not __debug__: + raise unittest.SkipTest( + '%s is disabled when __debug__ is disabled.' % cls.__name__) + super(DebugOnlyTestCase, cls).setUpClass() + + +class DeprecationTestCase(DebugOnlyTestCase, TestCase): """Test cases for deprecation function in the tools module.""" diff --git a/tests/deprecation_tests.py b/tests/deprecation_tests.py index bbdf02b..98f3ccc 100644 --- a/tests/deprecation_tests.py +++ b/tests/deprecation_tests.py @@ -10,7 +10,7 @@ from pywikibot.tools import ( deprecated, deprecate_arg, deprecated_args, add_full_name ) -from tests.aspects import unittest, DeprecationTestCase, TestCase +from tests.aspects import unittest, DeprecationTestCase @add_full_name @@ -53,7 +53,7 @@ pass -class DecoratorFullNameTestCase(TestCase): +class DecoratorFullNameTestCase(DeprecationTestCase): """Class with methods deprecated.""" @@ -286,7 +286,6 @@ self.assertDeprecation( __name__ + '.DeprecatedMethodClass.instance_method is deprecated.') - #(a)unittest.expectedFailure def test_deprecated_instance_method2(self): f = DeprecatedMethodClass() diff --git a/tests/dry_site_tests.py b/tests/dry_site_tests.py index 773888f..626cf7e 100644 --- a/tests/dry_site_tests.py +++ b/tests/dry_site_tests.py @@ -15,8 +15,9 @@ from pywikibot.exceptions import UnknownSite from tests.aspects import ( - unittest, TestCase, + unittest, DefaultDrySiteTestCase, + DebugOnlyTestCase, DeprecationTestCase, ) @@ -96,7 +97,7 @@ user_agent(x, format_string='Foo ({script_comments})')) -class TestMustBe(TestCase): +class TestMustBe(DebugOnlyTestCase): """Test cases for the must_be decorator.""" -- To view, visit https://gerrit.wikimedia.org/r/169788 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I67ed6b3e79ac46ef92e2b08ef6c606fd7177bae2 Gerrit-PatchSet: 9 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] update mw version - change (pywikibot/compat)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: update mw version ...................................................................... update mw version Change-Id: I7341f40c13b1e0206b1c8313a3a250334f2a722c --- M family.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/family.py b/family.py index c3e1b59..8b19856 100644 --- a/family.py +++ b/family.py @@ -4962,7 +4962,7 @@ # Don't use this, use versionnumber() instead. This only exists # to not break family files. # Here we return the latest mw release of wikimedia projects - return '1.25wmf5' + return '1.25wmf7' def shared_image_repository(self, code): return ('commons', 'commons') -- To view, visit https://gerrit.wikimedia.org/r/174380 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I7341f40c13b1e0206b1c8313a3a250334f2a722c Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/compat Gerrit-Branch: master Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] [IMPROV] Consistent usage of 'force' - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: [IMPROV] Consistent usage of 'force' ...................................................................... [IMPROV] Consistent usage of 'force' The newly added function _cache_interwikimap is using 'refresh' instead of 'force' and is the only function with this nomenclature. Change-Id: Ida651bfecc3fb51836376bd81e6e8fed384a1c71 --- M pywikibot/site.py 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Nullzero: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/site.py b/pywikibot/site.py index 4a139fa..f50d167 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -597,11 +597,11 @@ return [lang for lang in self.languages() if lang[:1].upper() + lang[1:] not in nsnames] - def _cache_interwikimap(self, refresh=False): + def _cache_interwikimap(self, force=False): """Cache the interwikimap with usable site instances.""" # _iw_sites is a local cache to return a APISite instance depending # on the interwiki prefix of that site - if refresh or not hasattr(self, '_iw_sites'): + if force or not hasattr(self, '_iw_sites'): self._iw_sites = {} for iw in self.siteinfo['interwikimap']: try: -- To view, visit https://gerrit.wikimedia.org/r/174028 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ida651bfecc3fb51836376bd81e6e8fed384a1c71 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Nullzero <nullzero.free(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] Stop login retrying if username is invalid - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Stop login retrying if username is invalid ...................................................................... Stop login retrying if username is invalid After an unsuccessful login using a password from the password file, when the API returns 'NotExists' or 'Invalid', repeating the login process with a new password supplied from the terminal by the user will not remedy the situation. Raise NoUsername exception instead, as the problem is a config issue. Bug: 73533 Change-Id: I4276c47be2e34fb6aba3490e2b029f9276b7d03a --- M pywikibot/exceptions.py M pywikibot/login.py 2 files changed, 32 insertions(+), 2 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve Nullzero: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/exceptions.py b/pywikibot/exceptions.py index b5013df..e711f48 100644 --- a/pywikibot/exceptions.py +++ b/pywikibot/exceptions.py @@ -3,7 +3,7 @@ Exception classes used throughout the framework. Error: Base class, all exceptions should the subclass of this class. - - NoUsername: Username is not in user-config.py + - NoUsername: Username is not in user-config.py, or it is invalid. - UserBlockedY: our username or IP has been blocked - AutoblockUser: requested action on a virtual autoblock user not valid - UserActionRefuse diff --git a/pywikibot/login.py b/pywikibot/login.py index ee05bd1..db7be8f 100644 --- a/pywikibot/login.py +++ b/pywikibot/login.py @@ -37,6 +37,22 @@ @deprecated_args(username="user", verbose=None) def __init__(self, password=None, sysop=False, site=None, user=None): + """ + Constructor. + + All parameters default to defaults in user-config. + + @param site: Site object to log into + @type site: BaseSite + @param user: username to use + @type user: basestring + @param password: password to use + @type password: basestring + @param sysop: login as sysop account + @type sysop: bool + + @raises NoUsername: No username is configured for the requested site. + """ if site is not None: self.site = site else: @@ -169,6 +185,14 @@ password_f.close() def login(self, retry=False): + """ + Attempt to log into the server. + + @param retry: infinitely retry if the API returns an unknown error + @type retry: bool + + @raises NoUsername: Username is not recognised by the site. + """ if not self.password: # As we don't want the password to appear on the screen, we set # password = True @@ -176,7 +200,6 @@ u'Password for user %(name)s on %(site)s (no characters will ' u'be shown):' % {'name': self.username, 'site': self.site}, password=True) -# self.password = self.password.encode(self.site.encoding()) pywikibot.output(u"Logging in to %(site)s as %(name)s" % {'name': self.username, 'site': self.site}) @@ -184,6 +207,13 @@ cookiedata = self.getCookie() except pywikibot.data.api.APIError as e: pywikibot.error(u"Login failed (%s)." % e.code) + if e.code == 'NotExists': + raise NoUsername(u"Username '%s' does not exist on %s" + % (self.username, self.site)) + elif e.code == 'Illegal': + raise NoUsername(u"Username '%s' is invalid on %s" + % (self.username, self.site)) + # TODO: investigate other unhandled API codes (bug 73539) if retry: self.password = None return self.login(retry=True) -- To view, visit https://gerrit.wikimedia.org/r/174035 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I4276c47be2e34fb6aba3490e2b029f9276b7d03a Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Nullzero <nullzero.free(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] Use envvar EDITOR if it exists, with tests - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Use envvar EDITOR if it exists, with tests ...................................................................... Use envvar EDITOR if it exists, with tests EDITOR is usually set on Unix platforms to be a basic editor. Use this as the default editor, if set, even on Windows. Set EDITOR in the script tests to be a noop (like /bin/true) so the script continues. Scripts 'editarticle' and 'upload' are now able to be tested. Change-Id: I8a069960d0919c760441538c0ac0d11d1015bf01 --- M pywikibot/config2.py M pywikibot/editor.py M tests/script_tests.py 3 files changed, 12 insertions(+), 8 deletions(-) Approvals: Merlijn van Deen: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/config2.py b/pywikibot/config2.py index e24ebca..3cf1608 100644 --- a/pywikibot/config2.py +++ b/pywikibot/config2.py @@ -337,6 +337,7 @@ # ############# EXTERNAL EDITOR SETTINGS ############## # The command for the editor you want to use. If set to None, a simple Tkinter # editor will be used. +editor = os.environ.get('EDITOR', None) # On Windows systems, this script tries to determine the default text editor. if sys.platform == 'win32': try: @@ -350,15 +351,13 @@ _key2 = _winreg.OpenKey(_winreg.HKEY_CLASSES_ROOT, '%s\shell\open\command' % _progID) _cmd = _winreg.QueryValueEx(_key2, None)[0] - editor = _cmd.replace('%1', '') + _cmd = _cmd.replace('%1', '') # Notepad is even worse than our Tkinter editor. - if editor.lower().endswith('notepad.exe'): - editor = None + if not _cmd.lower().endswith('notepad.exe'): + editor = _cmd except WindowsError: # Catch any key lookup errors - editor = None -else: - editor = None + pass # Warning: DO NOT use an editor which doesn't support Unicode to edit pages! # You will BREAK non-ASCII symbols! diff --git a/pywikibot/editor.py b/pywikibot/editor.py index 440dadc..0243717 100644 --- a/pywikibot/editor.py +++ b/pywikibot/editor.py @@ -54,7 +54,7 @@ command += " -n%i" % (line + 1) # seems not to support columns command += ' %s' % tempFilename - #print command + pywikibot.log(u'Running editor: %s' % command) return command def convertLinebreaks(self, text): diff --git a/tests/script_tests.py b/tests/script_tests.py index d575a66..e37cda9 100644 --- a/tests/script_tests.py +++ b/tests/script_tests.py @@ -55,7 +55,6 @@ unrunnable_script_list = [ 'script_wui', # depends on lua compiling - 'editarticle', # requires a X DISPLAY ] deadlock_script_list = [ @@ -211,7 +210,13 @@ else: return stream env = os.environ.copy() + # sys.path may have been modified by the test runner to load dependencies. env['PYTHONPATH'] = ":".join(sys.path) + # Set EDITOR to an executable that ignores all arguments and does nothing. + if sys.platform == 'win32': + env['EDITOR'] = 'call' + else: + env['EDITOR'] = 'true' options = { 'stdout': subprocess.PIPE, 'stderr': subprocess.PIPE -- To view, visit https://gerrit.wikimedia.org/r/172202 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8a069960d0919c760441538c0ac0d11d1015bf01 Gerrit-PatchSet: 3 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] Preloading tests - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Preloading tests ...................................................................... Preloading tests Adds tests for - PropertyGenerator, - APISite.preloadpages - PreloadingGenerator With several expected failures due to bugs with FIXME notes. Bug: 73461 Change-Id: I11df5361bd2b67cce132faba252552bfa5053827 --- M pywikibot/site.py M tests/api_tests.py M tests/pagegenerators_tests.py M tests/site_tests.py 4 files changed, 467 insertions(+), 20 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/site.py b/pywikibot/site.py index 799d99c..1242c70 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -2457,8 +2457,8 @@ break else: pywikibot.warning( - u"preloadpages: Query returned unexpected title" - u"'%s'" % pagedata['title']) + u"preloadpages: Query returned unexpected " + u"title '%s'" % pagedata['title']) continue except KeyError: pywikibot.debug(u"No 'title' in %s" % pagedata, _logger) diff --git a/tests/api_tests.py b/tests/api_tests.py index 5d6a07b..7c6c273 100644 --- a/tests/api_tests.py +++ b/tests/api_tests.py @@ -239,9 +239,9 @@ self.assertIn('revisions', pi.prefixes) -class TestPageGenerator(TestCase): +class TestDryPageGenerator(TestCase): - """API PageGenerator object test class.""" + """Dry API PageGenerator object test class.""" family = 'wikipedia' code = 'en' @@ -249,7 +249,7 @@ dry = True def setUp(self): - super(TestPageGenerator, self).setUp() + super(TestDryPageGenerator, self).setUp() mysite = self.get_site() self.gen = api.PageGenerator(site=mysite, generator="links", @@ -328,6 +328,134 @@ self.assertEqual(len(results), 4) # total=-1 but 4 expected +class TestPropertyGenerator(TestCase): + + """API PropertyGenerator object test class.""" + + family = 'wikipedia' + code = 'en' + + def test_info(self): + mainpage = self.get_mainpage() + links = list(self.site.pagelinks(mainpage, total=10)) + titles = [l.title(withSection=False) + for l in links] + gen = api.PropertyGenerator(site=self.site, + prop="info", + titles='|'.join(titles)) + + count = 0 + for pagedata in gen: + self.assertIsInstance(pagedata, dict) + self.assertIn('pageid', pagedata) + self.assertIn('lastrevid', pagedata) + count += 1 + self.assertEqual(len(links), count) + + def test_one_continuation(self): + mainpage = self.get_mainpage() + links = list(self.site.pagelinks(mainpage, total=10)) + titles = [l.title(withSection=False) + for l in links] + gen = api.PropertyGenerator(site=self.site, + prop="revisions", + titles='|'.join(titles)) + gen.set_maximum_items(-1) # suppress use of "rvlimit" parameter + + count = 0 + for pagedata in gen: + self.assertIsInstance(pagedata, dict) + self.assertIn('pageid', pagedata) + self.assertIn('revisions', pagedata) + self.assertIn('revid', pagedata['revisions'][0]) + count += 1 + self.assertEqual(len(links), count) + + def test_two_continuations(self): + mainpage = self.get_mainpage() + links = list(self.site.pagelinks(mainpage, total=10)) + titles = [l.title(withSection=False) + for l in links] + gen = api.PropertyGenerator(site=self.site, + prop="revisions|coordinates", + titles='|'.join(titles)) + gen.set_maximum_items(-1) # suppress use of "rvlimit" parameter + + count = 0 + for pagedata in gen: + self.assertIsInstance(pagedata, dict) + self.assertIn('pageid', pagedata) + self.assertIn('revisions', pagedata) + self.assertIn('revid', pagedata['revisions'][0]) + count += 1 + self.assertEqual(len(links), count) + + @unittest.expectedFailure + def test_many_continuations_limited(self): + mainpage = self.get_mainpage() + links = list(self.site.pagelinks(mainpage, total=30)) + titles = [l.title(withSection=False) + for l in links] + gen = api.PropertyGenerator(site=self.site, + prop="revisions|info|categoryinfo|langlinks|templates", + rvprop="ids|flags|timestamp|user|comment|content", + titles='|'.join(titles)) + + # An APIError is raised if set_maximum_items is not called. + gen.set_maximum_items(-1) # suppress use of "rvlimit" parameter + # Force the generator into continuation mode + gen.set_query_increment(5) + + count = 0 + for pagedata in gen: + self.assertIsInstance(pagedata, dict) + self.assertIn('pageid', pagedata) + count += 1 + self.assertEqual(len(links), count) + # FIXME: AssertionError: 30 != 6150 + + @unittest.expectedFailure + def test_two_continuations_limited(self): + # FIXME: test fails + mainpage = self.get_mainpage() + links = list(self.site.pagelinks(mainpage, total=30)) + titles = [l.title(withSection=False) + for l in links] + gen = api.PropertyGenerator(site=self.site, + prop="info|categoryinfo|langlinks|templates", + titles='|'.join(titles)) + # Force the generator into continuation mode + gen.set_query_increment(5) + + count = 0 + for pagedata in gen: + self.assertIsInstance(pagedata, dict) + self.assertIn('pageid', pagedata) + count += 1 + self.assertEqual(len(links), count) + # FIXME: AssertionError: 30 != 11550 + + # FIXME: test disabled as it takes longer than 10 minutes + def _test_two_continuations_limited_long_test(self): + """Long duration test, with total & step that are a real scenario.""" + mainpage = self.get_mainpage() + links = list(mainpage.backlinks(total=300)) + titles = [l.title(withSection=False) + for l in links] + gen = api.PropertyGenerator(site=self.site, + prop="info|categoryinfo|langlinks|templates", + titles='|'.join(titles)) + # Force the generator into continuation mode + gen.set_query_increment(50) + + count = 0 + for pagedata in gen: + self.assertIsInstance(pagedata, dict) + self.assertIn('pageid', pagedata) + count += 1 + self.assertEqual(len(links), count) + + class TestCachedRequest(DefaultSiteTestCase): """Test API Request caching. diff --git a/tests/pagegenerators_tests.py b/tests/pagegenerators_tests.py index 81720d1..9e19bde 100755 --- a/tests/pagegenerators_tests.py +++ b/tests/pagegenerators_tests.py @@ -14,7 +14,10 @@ import pywikibot from pywikibot import pagegenerators -from pywikibot.pagegenerators import PagesFromTitlesGenerator +from pywikibot.pagegenerators import ( + PagesFromTitlesGenerator, + PreloadingGenerator, +) from tests import _data_dir from tests.aspects import ( @@ -270,6 +273,41 @@ self.assertPagelistTitles(titles, self.expected_titles[site.case()]) +class TestPreloadingGenerator(DefaultSiteTestCase): + + """Test preloading generator on lists.""" + + def test_basic(self): + """Test PreloadingGenerator with a list of pages.""" + mainpage = self.get_mainpage() + links = list(self.site.pagelinks(mainpage, total=10)) + count = 0 + for page in PreloadingGenerator(links, step=20): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + self.assertEqual(len(links), count) + + def test_low_step(self): + """Test PreloadingGenerator with a list of pages.""" + mainpage = self.get_mainpage() + links = list(self.site.pagelinks(mainpage, total=20)) + count = 0 + for page in PreloadingGenerator(links, step=10): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + self.assertEqual(len(links), count) + + class TestDequePreloadingGenerator(DefaultSiteTestCase): """Test preloading generator on lists.""" diff --git a/tests/site_tests.py b/tests/site_tests.py index 408c60b..262bfd7 100644 --- a/tests/site_tests.py +++ b/tests/site_tests.py @@ -280,20 +280,6 @@ if a: self.assertEqual(a[0], mainpage) - def testPreload(self): - """Test that preloading works.""" - mysite = self.get_site() - mainpage = self.get_mainpage() - count = 0 - for page in mysite.preloadpages(mysite.pagelinks(mainpage, total=10)): - self.assertIsInstance(page, pywikibot.Page) - self.assertIsInstance(page.exists(), bool) - if page.exists(): - self.assertTrue(hasattr(page, "_text")) - count += 1 - if count >= 5: - break - def testLinkMethods(self): """Test site methods for getting links to and from a page.""" mysite = self.get_site() @@ -1562,6 +1548,301 @@ self.assertTrue(site.is_uploaddisabled()) +class TestPagePreloading(DefaultSiteTestCase): + + """Test site.preloadpages().""" + + def test_pageids(self): + """Test basic preloading with pageids.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = mysite.pagelinks(mainpage, total=10) + # preloadpages will send the page ids, + # as they have already been loaded by pagelinks + for page in mysite.preloadpages(links): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + if count >= 5: + break + + def test_titles(self): + """Test basic preloading with titles.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = mysite.pagelinks(mainpage, total=10) + + # remove the pageids that have already been loaded above by pagelinks + # so that preloadpages will use the titles instead + for page in links: + del page._pageid + + for page in mysite.preloadpages(links): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + if count >= 5: + break + + def test_preload_continuation(self): + """Test preloading continuation works.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = mysite.pagelinks(mainpage, total=10) + for page in mysite.preloadpages(links, groupsize=5): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + if count >= 6: + break + + def test_preload_high_groupsize(self): + """Test preloading continuation with groupsize greater than total.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + + # Determine if there are enough links on the main page, + # for the test to be useful. + link_count = len(list(mysite.pagelinks(mainpage, total=10))) + if link_count < 2: + raise unittest.SkipTest('insufficient links on main page') + + # get a fresh generator; we now know how many results it will have, + # if it is less than 10. + links = mysite.pagelinks(mainpage, total=10) + for page in mysite.preloadpages(links, groupsize=50): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + self.assertEqual(count, link_count) + + def test_preload_low_groupsize(self): + """Test preloading continuation with groupsize greater than total.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + + # Determine if there are enough links on the main page, + # for the test to be useful. + link_count = len(list(mysite.pagelinks(mainpage, total=10))) + if link_count < 2: + raise unittest.SkipTest('insufficient links on main page') + + # get a fresh generator; we now know how many results it will have, + # if it is less than 10. + links = mysite.pagelinks(mainpage, total=10) + for page in mysite.preloadpages(links, groupsize=5): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + self.assertEqual(count, link_count) + + def test_preload_unexpected_titles_using_pageids(self): + """Test sending pageids with unnormalized titles, causing warnings.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = list(mysite.pagelinks(mainpage, total=10)) + if len(links) < 2: + raise unittest.SkipTest('insufficient links on main page') + + # change the title of the page, to test sametitle(). + # preloadpages will send the page ids, as they have already been loaded + # by pagelinks, and preloadpages should complain the returned titles + # do not match any title in the pagelist. + for page in links: + page._link._text += ' ' + + gen = mysite.preloadpages(links, groupsize=5) + for page in gen: + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertFalse(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 0) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + if count > 5: + break + + def test_preload_unexpected_titles_using_titles(self): + """Test sending unnormalized titles, causing warnings.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = list(mysite.pagelinks(mainpage, total=10)) + if len(links) < 2: + raise unittest.SkipTest('insufficient links on main page') + + # change the title of the page _and_ delete the pageids. + # preloadpages can only send the titles, and preloadpages should + # complain the returned titles do not match any title in the pagelist. + for page in links: + page._link._text += ' ' + del page._pageid + + gen = mysite.preloadpages(links, groupsize=5) + for page in gen: + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertFalse(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 0) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + if count > 5: + break + + def test_preload_invalid_titles_without_pageids(self): + """Test sending invalid titles. No warnings issued, but it should.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = list(mysite.pagelinks(mainpage, total=10)) + if len(links) < 2: + raise unittest.SkipTest('insufficient links on main page') + + for page in links: + page._link._text += ' foobar' + del page._pageid + + gen = mysite.preloadpages(links, groupsize=5) + for page in gen: + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + self.assertFalse(page.exists()) + count += 1 + if count > 5: + break + + @unittest.expectedFailure + def test_preload_langlinks_normal(self): + """Test preloading continuation works.""" + # FIXME: test fails + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = mysite.pagelinks(mainpage, total=10) + for page in mysite.preloadpages(links, groupsize=5, langlinks=True): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + self.assertTrue(hasattr(page, '_langlinks')) + count += 1 + if count >= 6: + break + + @unittest.expectedFailure + def test_preload_langlinks_count(self): + """Test preloading continuation works.""" + # FIXME: test fails + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = mysite.pagelinks(mainpage, total=20) + pages = list(mysite.preloadpages(links, groupsize=5, + langlinks=True)) + for page in pages: + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + count += 1 + + self.assertEqual(len(links), count) + + def _test_preload_langlinks_long(self): + """Test preloading continuation works.""" + # FIXME: test fails. It is disabled as it takes more + # than 10 minutes on travis for English Wikipedia + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + links = mainpage.backlinks(total=100) + for page in mysite.preloadpages(links, groupsize=50, + langlinks=True): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + self.assertTrue(hasattr(page, '_langlinks')) + count += 1 + + self.assertEqual(len(links), count) + + @unittest.expectedFailure + def test_preload_templates(self): + """Test preloading templates works.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + # Use backlinks, as any backlink has at least one link + links = mysite.pagelinks(mainpage, total=10) + for page in mysite.preloadpages(links, templates=True): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + self.assertTrue(hasattr(page, '_templates')) + count += 1 + if count >= 6: + break + + @unittest.expectedFailure + def test_preload_templates_and_langlinks(self): + """Test preloading templates and langlinks works.""" + mysite = self.get_site() + mainpage = self.get_mainpage() + count = 0 + # Use backlinks, as any backlink has at least one link + links = mysite.pagebacklinks(mainpage, total=10) + for page in mysite.preloadpages(links, langlinks=True, templates=True): + self.assertIsInstance(page, pywikibot.Page) + self.assertIsInstance(page.exists(), bool) + if page.exists(): + self.assertTrue(hasattr(page, "_text")) + self.assertEqual(len(page._revisions), 1) + self.assertFalse(hasattr(page, '_pageprops')) + self.assertTrue(hasattr(page, '_templates')) + self.assertTrue(hasattr(page, '_langlinks')) + count += 1 + if count >= 6: + break + + class TestDataSitePreloading(WikidataTestCase): """Test DataSite.preloaditempages for repo pages.""" -- To view, visit https://gerrit.wikimedia.org/r/173499 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I11df5361bd2b67cce132faba252552bfa5053827 Gerrit-PatchSet: 5 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

[Gerrit] Page.categories parameter withSortKey ignored - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Page.categories parameter withSortKey ignored ...................................................................... Page.categories parameter withSortKey ignored The withSortKey parameter of Site.pagecategories has been marked as deprecated since the method was first introduced. However the method Page.categories has an equivalent parameter which it passes to Site.pagecategories, only for it to be ignored. Raise NotImplementedError if withSortKey is enabled calling Page.categories. Bug: 73561 Change-Id: I22ce77fb91c1f02b71dc028fe75178b1b40c1cde --- M pywikibot/page.py 1 file changed, 6 insertions(+), 2 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve Nullzero: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index bda181f..2d71d8a 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -1341,8 +1341,12 @@ @return: a generator that yields Category objects. """ - return self.site.pagecategories(self, withSortKey=withSortKey, - step=step, total=total, content=content) + # FIXME: bug 73561: withSortKey is ignored by Site.pagecategories + if withSortKey: + raise NotImplementedError('withSortKey is not implemented') + + return self.site.pagecategories(self, step=step, total=total, + content=content) def extlinks(self, step=None, total=None): """Iterate all external URLs (not interwiki links) from this page. -- To view, visit https://gerrit.wikimedia.org/r/174143 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I22ce77fb91c1f02b71dc028fe75178b1b40c1cde Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Nullzero <nullzero.free(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 5 months

← Newer
1
...
4
5
6
7
8
9
10
...
17
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits November 2014