Pywikibot-commits September 2014

pywikibot-commits@lists.wikimedia.org

1 participants
178 discussions

[Gerrit] Fix wikibase test test_own_client - change (pywikibot/core)
by jenkins-bot (Code Review) 04 Sep '14

04 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: Fix wikibase test test_own_client ...................................................................... Fix wikibase test test_own_client The test test_own_client was introduced when Wikidata became its own client. The test checked whether pywikibot could support this configuration, but it used an invalid test scenario to do this. It was introduced in dce03581d4a2f0e52aa5b7a2e2b1e68a5a1b48a1 The Main Page should not be a valid ItemPage title; that is being fixed in other changes. The intention was to see whether an item could be loaded for the normal page object for Main Page. Change-Id: I21686a13825f547cf1a4ea34dae697139f3e0bbe --- M tests/wikibase_tests.py 1 file changed, 6 insertions(+), 6 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve Ladsgroup: Looks good to me, approved jenkins-bot: Verified diff --git a/tests/wikibase_tests.py b/tests/wikibase_tests.py index d629e7e..9e1bca9 100644 --- a/tests/wikibase_tests.py +++ b/tests/wikibase_tests.py @@ -134,16 +134,16 @@ # working correctly on Wikidata. # The main Wikidata is its own client. - self.wdp = pywikibot.ItemPage(wikidata, - wikidata.siteinfo['mainpage']) + self.wdp = pywikibot.Page(wikidata, + wikidata.siteinfo['mainpage']) item = pywikibot.ItemPage.fromPage(self.wdp) - del item + self.assertEqual(item.site, self.wdp.site) # test.wikidata is also - self.wdp = pywikibot.ItemPage(wikidatatest, - wikidatatest.siteinfo['mainpage']) + self.wdp = pywikibot.Page(wikidatatest, + wikidatatest.siteinfo['mainpage']) item = pywikibot.ItemPage.fromPage(self.wdp) - del item + self.assertEqual(item.site, self.wdp.site) class TestItemLoad(PywikibotTestCase): -- To view, visit https://gerrit.wikimedia.org/r/158082 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I21686a13825f547cf1a4ea34dae697139f3e0bbe Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] [FIX] Make the user-agent Python 3 compatible - change (pywikibot/core)
by jenkins-bot (Code Review) 04 Sep '14

04 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: [FIX] Make the user-agent Python 3 compatible ...................................................................... [FIX] Make the user-agent Python 3 compatible Python 3 break two things about user-agent: - 'encode' returns bytes and not a string, because the encoded result isn't used and it's only important that it could be encoded the result is discarded - The version of the git directory can't be determined, because the stdout (and stderr/stdin) are bytes (list of bytes) and not strings. Those need to be converted. Change-Id: I191ab067eb2d8afe4e8f8d73f90408846add713a --- M pywikibot/comms/http.py M pywikibot/version.py 2 files changed, 7 insertions(+), 4 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py index 22a430d..e9703ff 100644 --- a/pywikibot/comms/http.py +++ b/pywikibot/comms/http.py @@ -143,7 +143,10 @@ return '' username = username.replace(' ', '_') # Avoid spaces or %20. try: - username = username.encode('ascii') + username.encode('ascii') # just test, but not actually use it + except UnicodeEncodeError: + pass + else: # % is legal in the default $wgLegalTitleChars # This is so that ops know the real pywikibot will not # allow a useragent in the username to allow through a hand-coded @@ -152,8 +155,6 @@ return quote(username) else: return username - except UnicodeEncodeError: - pass username = quote(username.encode('utf-8')) return username diff --git a/pywikibot/version.py b/pywikibot/version.py index b58fa2f..2fd8cf9 100644 --- a/pywikibot/version.py +++ b/pywikibot/version.py @@ -16,6 +16,8 @@ import urllib import subprocess +import pywikibot.config2 as config + cache = None @@ -206,7 +208,7 @@ '--date=iso'], cwd=_program_dir, stdout=subprocess.PIPE).stdout.read() - info = info.split('|') + info = info.decode(config.console_encoding).split('|') date = info[0][:-6] date = time.strptime(date.strip('"'), '%Y-%m-%d %H:%M:%S') rev = subprocess.Popen([cmd, 'rev-list', 'HEAD'], -- To view, visit https://gerrit.wikimedia.org/r/158318 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I191ab067eb2d8afe4e8f8d73f90408846add713a Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] [FEAT] If possible check if AssertEdit available - change (pywikibot/core)
by jenkins-bot (Code Review) 04 Sep '14

04 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: [FEAT] If possible check if AssertEdit available ...................................................................... [FEAT] If possible check if AssertEdit available Previously it assumed the AssertEdit extension is available, when the extensions weren't cached yet. But because a siteinfo request does not require an 'edit' action it should be safely possible to always retrieve the extensions information and always determine if it's installed or not. Change-Id: I5ed6a9c4921f5b453eca1eaeae8aa03baa5645a1 --- M pywikibot/data/api.py 1 file changed, 2 insertions(+), 6 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py index 8132a27..2839e5a 100644 --- a/pywikibot/data/api.py +++ b/pywikibot/data/api.py @@ -175,13 +175,9 @@ # otherwise be a problem. # This situation is only tripped when one of the first actions # on the site is a write action and the extension isn't installed. - if 'extensions' in self.site.siteinfo: - use_assert_edit_extension = self.site.has_extension('AssertEdit') - else: - use_assert_edit_extension = True - if ((self.write and LV(self.site.version()) >= LV("1.23")) or - (self.params["action"] == "edit" and use_assert_edit_extension)): + (self.params['action'] == 'edit' and + self.site.has_extension('AssertEdit'))): pywikibot.debug(u"Adding user assertion", _logger) self.params["assert"] = "user" # make sure user is logged in -- To view, visit https://gerrit.wikimedia.org/r/156462 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5ed6a9c4921f5b453eca1eaeae8aa03baa5645a1 Gerrit-PatchSet: 3 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] [FEAT] Cosmetic changes: Add option to ignore exceptions - change (pywikibot/core)
by jenkins-bot (Code Review) 04 Sep '14

04 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: [FEAT] Cosmetic changes: Add option to ignore exceptions ...................................................................... [FEAT] Cosmetic changes: Add option to ignore exceptions This feature allows to ignore exceptions and continue changing the current page or skipping it. Part of it (when ignoring a page) might be applicable to the bot class overall. Change-Id: I81094d94c32d012aba481a23eba8fe4db3dc5a17 --- M scripts/cosmetic_changes.py 1 file changed, 94 insertions(+), 33 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/cosmetic_changes.py b/scripts/cosmetic_changes.py index d11e09f..75d5c56 100755 --- a/scripts/cosmetic_changes.py +++ b/scripts/cosmetic_changes.py @@ -18,6 +18,9 @@ the predefined message texts with original and replacements inserted. +-ignore: Ignores if an error occured and either skips the page or + only that method. It can be set to 'page' or 'method'. + &warning; For regular use, it is recommended to put this line into your user-config.py: @@ -147,9 +150,15 @@ } +CANCEL_ALL = False +CANCEL_PAGE = 1 +CANCEL_METHOD = 2 + + class CosmeticChangesToolkit: + def __init__(self, site, debug=False, redirect=False, namespace=None, - pageTitle=None): + pageTitle=None, ignore=CANCEL_ALL): self.site = site self.debug = debug self.redirect = redirect @@ -157,40 +166,77 @@ self.template = (self.namespace == 10) self.talkpage = self.namespace >= 0 and self.namespace % 2 == 1 self.title = pageTitle + self.ignore = ignore - def change(self, text): - """Given a wiki source code text, return the cleaned up version.""" - oldText = text - if self.site.sitename() == u'commons:commons' and self.namespace == 6: - text = self.commonsfiledesc(text) - text = self.fixSelfInterwiki(text) - text = self.standardizePageFooter(text) - text = self.fixSyntaxSave(text) - text = self.cleanUpLinks(text) - text = self.cleanUpSectionHeaders(text) - text = self.putSpacesInLists(text) - text = self.translateAndCapitalizeNamespaces(text) -## text = self.translateMagicWords(text) - text = self.replaceDeprecatedTemplates(text) -## text = self.resolveHtmlEntities(text) - text = self.validXhtml(text) - text = self.removeUselessSpaces(text) - text = self.removeNonBreakingSpaceBeforePercent(text) + self.common_methods = ( + self.commonsfiledesc, + self.fixSelfInterwiki, + self.standardizePageFooter, + self.fixSyntaxSave, + self.cleanUpLinks, + self.cleanUpSectionHeaders, + self.putSpacesInLists, + self.translateAndCapitalizeNamespaces, +## self.translateMagicWords, + self.replaceDeprecatedTemplates, +## self.resolveHtmlEntities, + self.validXhtml, + self.removeUselessSpaces, + self.removeNonBreakingSpaceBeforePercent, - text = self.fixHtml(text) - text = self.fixReferences(text) - text = self.fixStyle(text) - text = self.fixTypo(text) - if self.site.code in ['ckb', 'fa']: - text = self.fixArabicLetters(text) + self.fixHtml, + self.fixReferences, + self.fixStyle, + self.fixTypo, + + self.fixArabicLetters, + ) + + def safe_execute(self, method, text): + """Execute the method and catch exceptions if enabled.""" + result = None try: - text = isbn.hyphenateIsbnNumbers(text) + result = method(text) + except Exception as e: + if self.ignore == CANCEL_METHOD: + pywikibot.warning(u'Unable to perform "{0}" on "{1}"!'.format( + method.__name__, self.title)) + pywikibot.exception(e) + else: + raise + return text if result is None else text + + @staticmethod + def isbn_execute(text): + """Hyphenate ISBN numbers and catch 'InvalidIsbnException'.""" + try: + return isbn.hyphenateIsbnNumbers(text) except isbn.InvalidIsbnException as error: pywikibot.log(u"ISBN error: %s" % error) - pass - if self.debug: - pywikibot.showDiff(oldText, text) + return None + + def _change(self, text): + """Execute all clean up methods.""" + for method in self.common_methods: + text = self.safe_execute(method, text) + text = self.safe_execute(CosmeticChangesToolkit.isbn_execute, text) return text + + def change(self, text): + """Execute all clean up methods and catch errors if activated.""" + try: + new_text = self._change(text) + except Exception as e: + if self.ignore == CANCEL_PAGE: + pywikibot.warning(u'Skipped "{0}", because an error occured.'.format(self.title)) + pywikibot.exception(e) + return False + else: + raise + else: + if self.debug: + pywikibot.showDiff(text, new_text) + return new_text def fixSelfInterwiki(self, text): """ @@ -722,6 +768,8 @@ return text def fixArabicLetters(self, text): + if self.site.code not in ['ckb', 'fa']: + return exceptions = [ 'gallery', 'hyperlink', @@ -784,6 +832,8 @@ # Retrieved from "https://commons.wikimedia.org/wiki/Commons:Tools/pywiki_file_description_cl…" def commonsfiledesc(self, text): + if self.site.sitename() != u'commons:commons' or self.namespace == 6: + return # section headers to {{int:}} versions exceptions = ['comment', 'includeonly', 'math', 'noinclude', 'nowiki', 'pre', 'source', 'ref', 'timeline'] @@ -838,6 +888,7 @@ self.availableOptions.update({ 'async': False, 'comment': u'Robot: Cosmetic changes', + 'ignore': CANCEL_ALL, }) super(CosmeticChangesBot, self).__init__(**kwargs) @@ -848,11 +899,13 @@ self.current_page = page ccToolkit = CosmeticChangesToolkit(page.site, debug=True, namespace=page.namespace(), - pageTitle=page.title()) + pageTitle=page.title(), + ignore=self.getOption('ignore')) changedText = ccToolkit.change(page.get()) - self.userPut(page, page.text, changedText, - comment=self.getOption('comment'), - async=self.getOption('async')) + if changedText is not False: + self.userPut(page, page.text, changedText, + comment=self.getOption('comment'), + async=self.getOption('async')) except pywikibot.NoPage: pywikibot.output("Page %s does not exist?!" % page.title(asLink=True)) @@ -881,6 +934,14 @@ options['always'] = True elif arg == '-async': options['async'] = True + elif arg.startswith('-ignore:'): + ignore_mode = arg[len('-ignore:'):].lower() + if ignore_mode == 'method': + options['ignore'] = CANCEL_METHOD + elif ignore_mode == 'page': + options['ignore'] = CANCEL_PAGE + else: + raise ValueError('Unknown ignore mode "{0}"!'.format(ignore_mode)) else: genFactory.handleArg(arg) -- To view, visit https://gerrit.wikimedia.org/r/158134 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I81094d94c32d012aba481a23eba8fe4db3dc5a17 Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] MVP: implement Echo support - change (pywikibot/core)
by jenkins-bot (Code Review) 03 Sep '14

03 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: MVP: implement Echo support ...................................................................... MVP: implement Echo support with a new Notification class in echo.py and new APISite methods: - notifications() - notifications_mark_read() bug: 66275 Change-Id: I25d72ce5f3f3418d8f92e9e41186aaf07fef04ea --- A pywikibot/echo.py M pywikibot/site.py 2 files changed, 84 insertions(+), 0 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/echo.py b/pywikibot/echo.py new file mode 100644 index 0000000..a7ed138 --- /dev/null +++ b/pywikibot/echo.py @@ -0,0 +1,48 @@ +# -*- coding: utf-8 -*- +"""Classes and functions for working with the Echo extension.""" + +import pywikibot + + +class Notification(object): + + """A notification issued by the Echo extension.""" + + def __init__(self, site): + """Construct an empty Notification object.""" + self.site = site + + @classmethod + def fromJSON(cls, site, data): + """ + Construct a Notification object from JSON data returned by the API. + + @rtype: Notification + """ + notif = cls(site) + + notif.id = data['id'] # TODO: use numeric id ? + notif.type = data['type'] + notif.category = data['category'] + notif.timestamp = pywikibot.Timestamp.fromtimestampformat(data['timestamp']['mw']) + + # TODO: use 'namespace-key' + 'text' ? + notif.page = pywikibot.Page(site, data['title']['full']) + + if 'agent' in data and 'name' in data['agent']: + notif.agent = pywikibot.User(site, data['agent']['name']) + else: + notif.agent = None + + if 'read' in data: + notif.read = pywikibot.Timestamp.fromtimestampformat(data['read']) + else: + notif.read = False + + notif.content = data.get('*', None) + + return notif + + def mark_as_read(self): + """Mark the notification as read.""" + return self.site.notifications_mark_read(list=self.id) diff --git a/pywikibot/site.py b/pywikibot/site.py index 8e42f27..e5da073 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -42,6 +42,7 @@ SpamfilterError, UserBlocked, ) +from pywikibot.echo import Notification if sys.version_info[0] > 2: basestring = (str,) @@ -1480,6 +1481,41 @@ self.login(sysop) return 'hasmsg' in self._userinfo + def notifications(self, **kwargs): + """Yield Notification objects from the Echo extension.""" + if self.has_extension('Echo'): + params = dict(site=self, action='query', + meta='notifications', + notprop='list', notformat='text') + + for key in kwargs: + params['not' + key] = kwargs[key] + + data = api.Request(**params).submit() + for notif in data['query']['notifications']['list'].values(): + yield Notification.fromJSON(self, notif) + + def notifications_mark_read(self, **kwargs): + """Mark one, some or all notifications, + selected via keyword arguments, as read. + + @return: whether the action was successful + @rtype: bool + """ + if self.has_extension('Echo'): + # TODO: ensure that the 'echomarkread' action + # is supported by the site + req = api.Request(site=self, + action='echomarkread', + token=self.token(pywikibot.Page(self, u'Main Page'), + 'edit'), # Use a dummy page + **kwargs) + data = req.submit() + try: + return data['query']['echomarkread']['result'] == 'success' + except KeyError: + return False + def mediawiki_messages(self, keys): """Fetch the text of a set of MediaWiki messages. -- To view, visit https://gerrit.wikimedia.org/r/138045 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I25d72ce5f3f3418d8f92e9e41186aaf07fef04ea Gerrit-PatchSet: 10 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Ricordisamoa <ricordisamoa(a)openmailbox.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Multichill <maarten(a)mdammers.nl> Gerrit-Reviewer: Ricordisamoa <ricordisamoa(a)openmailbox.org> Gerrit-Reviewer: Russell Blau <russblau(a)imapmail.org> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] Use page properties to find if a category is hidden - change (pywikibot/core)
by jenkins-bot (Code Review) 03 Sep '14

03 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: Use page properties to find if a category is hidden ...................................................................... Use page properties to find if a category is hidden Bug: 65149 Added also: @pywikibot.site.need_version("1.13") def isEmptyCategory(self): @pywikibot.site.need_version("1.11") def isHiddenCategory(self): New method Page.version() defined in order to use such decorator. Change-Id: I349559ee93fae1c6f8832581ed774da965740539 --- M pywikibot/page.py M tests/page_tests.py 2 files changed, 19 insertions(+), 6 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index 9a865a8..1f1390b 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -120,6 +120,15 @@ """Return the Site object for the wiki on which this Page resides.""" return self._link.site + def version(self): + """Return MediaWiki version number of the Site object for the wiki + on which this Page resides. + + This is needed to use @need_version() decorator for methods of + Page objects. + """ + return self.site.version() + @property def image_repository(self): """Return the Site object for the image repository.""" @@ -2119,19 +2128,16 @@ if total == 0: return + @pywikibot.site.need_version("1.13") def isEmptyCategory(self): """Return True if category has no members (including subcategories).""" ci = self.categoryinfo return sum(ci[k] for k in ['files', 'pages', 'subcats']) == 0 + @pywikibot.site.need_version("1.11") def isHiddenCategory(self): """Return True if the category is hidden.""" - # FIXME - # This should use action=query&list=allcategories - # setting acfrom and acto to the category title and adding - # acprop=hidden but currently fails in some cases - # (see bug 48824) - return '__HIDDENCAT__' in self.expand_text() + return u'hiddencat' in self.properties() def copyTo(self, cat, message): """ diff --git a/tests/page_tests.py b/tests/page_tests.py index 4383cb1..d8590c9 100644 --- a/tests/page_tests.py +++ b/tests/page_tests.py @@ -410,6 +410,13 @@ self.assertTrue(cat_empty.isEmptyCategory()) self.assertFalse(cat_not_empty.isEmptyCategory()) + def test_isHiddenCategory(self): + site = pywikibot.Site('en', 'wikipedia') + cat_hidden = pywikibot.Category(site, u'Category:Hidden categories') + cat_not_hidden = pywikibot.Category(site, u'Category:Wikipedia categories') + self.assertTrue(cat_hidden.isHiddenCategory()) + self.assertFalse(cat_not_hidden.isHiddenCategory()) + if __name__ == '__main__': try: -- To view, visit https://gerrit.wikimedia.org/r/157281 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I349559ee93fae1c6f8832581ed774da965740539 Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Nullzero <nullzero.free(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] User-agent graceful degradation - change (pywikibot/core)
by jenkins-bot (Code Review) 03 Sep '14

03 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: User-agent graceful degradation ...................................................................... User-agent graceful degradation The newly formatted user-agent string is not compliant with RFC 7231, as it puts the username at the end, when it should be at the beginning and leaves '(User:)' in the user-agent if the request was not to a site. Using a default format string of '{script}/{version} ..' is asking for the end-user to build a uncompliant user-agent. We should give them properly formatted products and product comments. Provide Pywikibot, httplib2 and Python release versions as products, and provide script_product and script_comments, populated with the information currently known about the script. In the user-agent builder, include in the user-agent whichever components are available at the time of the request, allowing http.py to be used by version.py without cyclic dependency runtime exceptions. Avoid percent-encoding or utf-8 encoding the username unless necessary. Bug: 55016 Bug: 66102 Change-Id: Iedf28f5e0a216ba23a015924e564ae0537e64f0d --- M pywikibot/comms/http.py M pywikibot/config2.py M tests/dry_site_tests.py M tests/http_tests.py 4 files changed, 187 insertions(+), 21 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve XZise: Looks good to me, but someone else must approve Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/comms/http.py b/pywikibot/comms/http.py index 564026a..22a430d 100644 --- a/pywikibot/comms/http.py +++ b/pywikibot/comms/http.py @@ -123,6 +123,85 @@ # export cookie_jar to global namespace pywikibot.cookie_jar = cookie_jar +USER_AGENT_PRODUCTS = { + 'python': 'Python/' + '.'.join([str(i) for i in sys.version_info]), + 'httplib2': 'httplib2/' + httplib2.__version__, + 'pwb': 'Pywikibot/' + pywikibot.__release__, +} + + +def user_agent_username(username=None): + """ + Reduce username to a representation permitted in HTTP headers. + + To achieve that, this function: + 1) replaces spaces (' ') with '_' + 2) encodes the username as 'utf-8' and if the username is not ASCII + 3) URL encodes the username if it is not ASCII, or contains '%' + """ + if not username: + return '' + username = username.replace(' ', '_') # Avoid spaces or %20. + try: + username = username.encode('ascii') + # % is legal in the default $wgLegalTitleChars + # This is so that ops know the real pywikibot will not + # allow a useragent in the username to allow through a hand-coded + # percent-encoded value. + if '%' in username: + return quote(username) + else: + return username + except UnicodeEncodeError: + pass + username = quote(username.encode('utf-8')) + return username + + +def user_agent(site=None, format_string=None): + values = USER_AGENT_PRODUCTS.copy() + + # This is the Pywikibot revision; also map it to {version} at present. + if pywikibot.version.cache: + values['revision'] = pywikibot.version.cache['rev'] + else: + values['revision'] = '' + values['version'] = values['revision'] + + values['script'] = pywikibot.calledModuleName() + + # TODO: script_product should add the script version, if known + values['script_product'] = pywikibot.calledModuleName() + + script_comments = [] + username = '' + if site: + script_comments.append(str(site)) + + # TODO: there are several ways of identifying a user, and username + # is not the best for a HTTP header if the username isnt ASCII. + if site.username(): + username = user_agent_username(site.username()) + script_comments.append( + 'User:' + username) + + values.update({ + 'family': site.family.name if site else '', + 'code': site.code if site else '', + 'lang': site.code if site else '', # TODO: use site.lang, if known + 'site': str(site) if site else '', + 'username': username, + 'script_comments': '; '.join(script_comments) + }) + + if not format_string: + format_string = config.user_agent_format + + formatted = format_string.format(**values) + # clean up after any blank components + formatted = formatted.replace(u'()', u'').replace(u' ', u' ').strip() + return formatted + def request(site, uri, ssl=False, *args, **kwargs): """Queue a request to be submitted to Site. @@ -151,24 +230,10 @@ baseuri = urlparse.urljoin("%s://%s" % (proto, host), uri) else: baseuri = uri - if "headers" not in kwargs: - kwargs["headers"] = {} - if site: - username = site.username() - if not username: - username = "" - kwargs["headers"]["user-agent"] = config.USER_AGENT_FORMAT.format( - script=pywikibot.calledModuleName(), - version=pywikibot.version.getversiondict()['rev'], - username=quote(username.encode('utf-8')), - lang=site.code, - family=site.family.name) - else: - USER_AGENT_FORMAT = '{script}/{version} Pywikibot/2.0' - kwargs["headers"]["user-agent"] = USER_AGENT_FORMAT.format( - script=pywikibot.calledModuleName(), - version=pywikibot.version.getversiondict()['rev']) + format_string = kwargs.setdefault("headers", {}).get("user-agent") + kwargs["headers"]["user-agent"] = user_agent(site, format_string) + request = threadedhttp.HttpRequest(baseuri, *args, **kwargs) http_queue.put(request) while not request.lock.acquire(False): diff --git a/pywikibot/config2.py b/pywikibot/config2.py index 6ea3c45..0fe5f38 100644 --- a/pywikibot/config2.py +++ b/pywikibot/config2.py @@ -76,9 +76,7 @@ # User agent format. # For the meaning and more help in customization see: # https://www.mediawiki.org/wiki/Manual:Pywikibot/User-agent -# The default is script, revision number and user name -# For more information see https://meta.wikimedia.org/wiki/User-agent_policy -USER_AGENT_FORMAT = '{script}/{version} Pywikibot/2.0 (User:{username})' +user_agent_format = '{script_product} ({script_comments}) {pwb} ({revision}) {httplib2} {python}' # The default interface for communicating with the site # currently the only defined interface is 'APISite', so don't change this! diff --git a/tests/dry_site_tests.py b/tests/dry_site_tests.py index ff75020..c3e12aa 100644 --- a/tests/dry_site_tests.py +++ b/tests/dry_site_tests.py @@ -9,6 +9,7 @@ import pywikibot from pywikibot.site import must_be, need_version +from pywikibot.comms.http import user_agent from tests.utils import unittest, NoSiteTestCase, DummySiteinfo @@ -44,6 +45,56 @@ self.assertTrue(x.logged_in(True)) self.assertFalse(x.logged_in(False)) + def test_user_agent(self): + x = DrySite('en', 'wikipedia') + + x._userinfo = {'name': 'foo'} + x._username = ('foo', None) + + self.assertEqual('Pywikibot/' + pywikibot.__release__, + user_agent(x, format_string='{pwb}')) + + self.assertEqual(x.family.name, + user_agent(x, format_string='{family}')) + self.assertEqual(x.code, + user_agent(x, format_string='{lang}')) + self.assertEqual(x.family.name + ' ' + x.code, + user_agent(x, format_string='{family} {lang}')) + + self.assertEqual(x.username(), + user_agent(x, format_string='{username}')) + + x._userinfo = {'name': u'!'} + x._username = (u'!', None) + + self.assertEqual('!', user_agent(x, format_string='{username}')) + + x._userinfo = {'name': u'foo bar'} + x._username = (u'foo bar', None) + + self.assertEqual('foo_bar', user_agent(x, format_string='{username}')) + + old_config = '{script}/{version} Pywikibot/2.0 (User:{username})' + + pywikibot.version.getversiondict() + script_value = pywikibot.calledModuleName() + '/' + pywikibot.version.cache['rev'] + + self.assertEqual(script_value + ' Pywikibot/2.0 (User:foo_bar)', + user_agent(x, format_string=old_config)) + + x._userinfo = {'name': u'⁂'} + x._username = (u'⁂', None) + + self.assertEqual('%E2%81%82', + user_agent(x, format_string='{username}')) + + x._userinfo = {'name': u'127.0.0.1'} + x._username = (None, None) + + self.assertEqual('Foo', user_agent(x, format_string='Foo {username}')) + self.assertEqual('Foo (wikipedia:en)', + user_agent(x, format_string='Foo ({script_comments})')) + class TestMustBe(NoSiteTestCase): diff --git a/tests/http_tests.py b/tests/http_tests.py index e139fbb..9229cdd 100644 --- a/tests/http_tests.py +++ b/tests/http_tests.py @@ -7,8 +7,10 @@ # __version__ = '$Id$' - +import sys +import pywikibot from pywikibot.comms import http, threadedhttp +from pywikibot import config2 as config from tests.utils import unittest, NoSiteTestCase @@ -45,6 +47,56 @@ self.assertIn('-content-encoding', r[0]) self.assertEqual(r[0]['-content-encoding'], 'gzip') + def test_user_agent(self): + self.assertEqual('', http.user_agent(format_string=' ')) + self.assertEqual('', http.user_agent(format_string=' ')) + self.assertEqual('a', http.user_agent(format_string=' a ')) + + # if there is no site, these can't have a value + self.assertEqual('', http.user_agent(format_string='{username}')) + self.assertEqual('', http.user_agent(format_string='{family}')) + self.assertEqual('', http.user_agent(format_string='{lang}')) + + self.assertEqual('Pywikibot/' + pywikibot.__release__, + http.user_agent(format_string='{pwb}')) + self.assertNotIn(' ', http.user_agent(format_string=' {pwb} ')) + + self.assertIn('Pywikibot/' + pywikibot.__release__, + http.user_agent(format_string='SVN/1.7.5 {pwb}')) + + def test_user_agent_username(self): + self.assertEqual('%25', http.user_agent_username('%')) + self.assertEqual('%2525', http.user_agent_username('%25')) + self.assertEqual(';', http.user_agent_username(';')) + self.assertEqual('-', http.user_agent_username('-')) + self.assertEqual('.', http.user_agent_username('.')) + self.assertEqual("'", http.user_agent_username("'")) + self.assertEqual('foo_bar', http.user_agent_username('foo bar')) + + self.assertEqual('%E2%81%82', http.user_agent_username(u'⁂')) + + +class DefaultUserAgentTestCase(NoSiteTestCase): + + def setUp(self): + self.orig_format = config.user_agent_format + config.user_agent_format = '{script_product} ({script_comments}) {pwb} ({revision}) {httplib2} {python}' + + def tearDown(self): + config.user_agent_format = self.orig_format + + def test_default_user_agent(self): + """ Config defined format string test. """ + self.assertTrue(http.user_agent().startswith( + pywikibot.calledModuleName())) + self.assertIn('Pywikibot/' + pywikibot.__release__, http.user_agent()) + self.assertNotIn(' ', http.user_agent()) + self.assertNotIn('()', http.user_agent()) + self.assertNotIn('(;', http.user_agent()) + self.assertNotIn(';)', http.user_agent()) + self.assertIn('httplib2/', http.user_agent()) + self.assertIn('Python/' + str(sys.version_info[0]), http.user_agent()) + if __name__ == '__main__': try: -- To view, visit https://gerrit.wikimedia.org/r/152200 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iedf28f5e0a216ba23a015924e564ae0537e64f0d Gerrit-PatchSet: 13 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Nullzero <nullzero.free(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] Bug 69039 - welcome raises exception if site is not configured - change (pywikibot/core)
by jenkins-bot (Code Review) 03 Sep '14

03 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: Bug 69039 - welcome raises exception if site is not configured ...................................................................... Bug 69039 - welcome raises exception if site is not configured welcome.py checks if site is managed by the script, otherwise exit without error is site is not managed by the script and show help. Change-Id: I8a7ce9a6442841a897ec2e4854f5e80bfe5c781f --- M scripts/welcome.py 1 file changed, 31 insertions(+), 2 deletions(-) Approvals: John Vandenberg: Looks good to me, approved XZise: Looks good to me, but someone else must approve jenkins-bot: Verified diff --git a/scripts/welcome.py b/scripts/welcome.py index 529984e..0456bf6 100644 --- a/scripts/welcome.py +++ b/scripts/welcome.py @@ -433,9 +433,13 @@ class WelcomeBot(object): + """Bot to add welcome messages on User pages.""" + def __init__(self): - #Initial + """Constructor.""" + self.site = pywikibot.Site() + self.check_managed_sites() self.bname = dict() self._totallyCount = 0 @@ -445,6 +449,23 @@ self.defineSign(True) if __name__ != '__main__': # use only in module call self._checkQueue = [] + + def check_managed_sites(self): + """Check that site is managed by welcome.py.""" + # Raises KeyError if site is not in netext with right family. + try: + site_netext = netext[self.site.family.name] + except KeyError: + raise KeyError( + u'Site %s not managed by welcome.py: family "%s" missing in netext.' + % (self.site, self.site.family.name)) + # Raises KeyError if site is not in netext with language. + try: + site_netext = site_netext[self.site.code] + except KeyError: + raise KeyError( + u'Site %s not managed by welcome.py: lang "%s" missing in netext[%s].' + % (self.site, self.site.code, self.site.family.name)) def badNameFilter(self, name, force=False): if not globalvar.filtBadName: @@ -950,7 +971,15 @@ pywikibot.warning( 'both -offset and -timeoffset were provided, ignoring -offset') globalvar.offset = 0 - bot = WelcomeBot() + + try: + bot = WelcomeBot() + except KeyError as error: + # site not managed by welcome.py + pywikibot.showHelp() + pywikibot.warning(error) + return + try: bot.run() except KeyboardInterrupt: -- To view, visit https://gerrit.wikimedia.org/r/157983 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8a7ce9a6442841a897ec2e4854f5e80bfe5c781f Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] [FIX] Bot: The unicode title must be used when logging it. - change (pywikibot/core)
by jenkins-bot (Code Review) 03 Sep '14

03 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: [FIX] Bot: The unicode title must be used when logging it. ...................................................................... [FIX] Bot: The unicode title must be used when logging it. Change-Id: Ieced102c74a0795be118817ef8b4eb45c1f274c9 --- M pywikibot/bot.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/bot.py b/pywikibot/bot.py index b332099..614814f 100644 --- a/pywikibot/bot.py +++ b/pywikibot/bot.py @@ -857,7 +857,7 @@ """ if page != self._current_page: self._current_page = page - msg = u'Working on %r' % page + msg = u'Working on %r' % page.title() if config.colorized_output: log(msg) stdout(u'\n\n>>> \03{lightpurple}%s\03{default} <<<' -- To view, visit https://gerrit.wikimedia.org/r/158096 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ieced102c74a0795be118817ef8b4eb45c1f274c9 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

1 0

[Gerrit] [FIX] Cosmetic_changes: Use 'comment' option not attribute - change (pywikibot/core)
by jenkins-bot (Code Review) 03 Sep '14

03 Sep '14

jenkins-bot has submitted this change and it was merged. Change subject: [FIX] Cosmetic_changes: Use 'comment' option not attribute ...................................................................... [FIX] Cosmetic_changes: Use 'comment' option not attribute Change-Id: If7477a22961f4cc54e39bac2d82b6edb920a27c9 --- M scripts/cosmetic_changes.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Ricordisamoa: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/cosmetic_changes.py b/scripts/cosmetic_changes.py index 9e87462..d11e09f 100755 --- a/scripts/cosmetic_changes.py +++ b/scripts/cosmetic_changes.py @@ -850,7 +850,8 @@ namespace=page.namespace(), pageTitle=page.title()) changedText = ccToolkit.change(page.get()) - self.userPut(page, page.text, changedText, comment=self.comment, + self.userPut(page, page.text, changedText, + comment=self.getOption('comment'), async=self.getOption('async')) except pywikibot.NoPage: pywikibot.output("Page %s does not exist?!" -- To view, visit https://gerrit.wikimedia.org/r/158094 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: If7477a22961f4cc54e39bac2d82b6edb920a27c9 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Ricordisamoa <ricordisamoa(a)openmailbox.org> Gerrit-Reviewer: jenkins-bot <>

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits September 2014