Pywikibot-commits April 2017

pywikibot-commits@lists.wikimedia.org

2 participants
100 discussions

[Gerrit] pywikibot/core[master]: Fix spelling mistake

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/347308 ) Change subject: Fix spelling mistake ...................................................................... Fix spelling mistake Also keep next line beneath 80 chars. Change-Id: I8894eb8a5129f7b99e8ad610252d189a4aea5fbd --- M pywikibot/proofreadpage.py 1 file changed, 3 insertions(+), 2 deletions(-) Approvals: Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/proofreadpage.py b/pywikibot/proofreadpage.py index 4034cb5..47bf1ab 100644 --- a/pywikibot/proofreadpage.py +++ b/pywikibot/proofreadpage.py @@ -473,8 +473,9 @@ and relative page numbers and labels by means of several helper functions. - It also providesa generator to pages contained in Index page, with - possibility to define range, filter by quality levels and page existance. + It also provides a generator to pages contained in Index page, with + possibility to define range, filter by quality levels and page + existence. @raise UnknownExtension: source Site has no ProofreadPage Extension. @raise ImportError: bs4 is not installed. -- To view, visit https://gerrit.wikimedia.org/r/347308 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I8894eb8a5129f7b99e8ad610252d189a4aea5fbd Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: Only require that all encuntered data types are known

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/347336 ) Change subject: Only require that all encuntered data types are known ...................................................................... Only require that all encuntered data types are known This changes the requirements of the test to require that all encountered data types are known but not that all known data types are encountered. Note that test_datatypes is still expected to fail until T161726 is resolved, BUT it should no longer fail afterwards. Bug: T160339 Change-Id: I6e1e4cf0194dc786c845c50a5d7b3fe651c6dce8 --- M tests/paraminfo_tests.py 1 file changed, 10 insertions(+), 4 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/tests/paraminfo_tests.py b/tests/paraminfo_tests.py index 2899314..f1162f7 100644 --- a/tests/paraminfo_tests.py +++ b/tests/paraminfo_tests.py @@ -41,14 +41,19 @@ return param['type'] def _check_param_values(self, site, module, parameter, expected): - """Perform check that a parameter matches the expected list.""" + """Check that a parameter matches the expected list exactly.""" values = self._get_param_values(site, module, parameter) self.assertCountEqual(expected, values) def _check_param_subset(self, site, module, parameter, expected): - """Perform check that a parameter matches the expected list.""" + """Check that a parameter contains all entries in expected list.""" values = self._get_param_values(site, module, parameter) self.assertLessEqual(set(expected), set(values)) + + def _check_param_superset(self, site, module, parameter, expected): + """Check that a parameter only contains entries in expected list.""" + values = self._get_param_values(site, module, parameter) + self.assertGreaterEqual(set(expected), set(values)) class MediaWikiKnownTypesTestCase(KnownTypesTestBase, @@ -226,10 +231,11 @@ self._check_param_values(self.repo, 'wbsearchentities', 'type', known) def test_datatypes(self): - """Test known datatypes.""" + """Test that all encountered datatypes are known.""" unsupported = set(['wikibase-property']) known = set(Property.types) | unsupported - self._check_param_values(self.repo, 'wbformatvalue', 'datatype', known) + self._check_param_superset( + self.repo, 'wbformatvalue', 'datatype', known) def test_snaktype(self): """Test known snak types.""" -- To view, visit https://gerrit.wikimedia.org/r/347336 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I6e1e4cf0194dc786c845c50a5d7b3fe651c6dce8 Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Lokal Profil <lokal.profil(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Lokal Profil <lokal.profil(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: Correct copy-pasted docstring

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/347333 ) Change subject: Correct copy-pasted docstring ...................................................................... Correct copy-pasted docstring Change-Id: Ida681e2a71d261ef3afcf34387a29388655856a7 --- M tests/edit_failure_tests.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/tests/edit_failure_tests.py b/tests/edit_failure_tests.py index f31a3b9..5c8fcb9 100644 --- a/tests/edit_failure_tests.py +++ b/tests/edit_failure_tests.py @@ -174,7 +174,7 @@ item.addClaim, claim) def test_url_invalid_protocol(self): - """Attempt adding invalid latex to a math claim.""" + """Attempt adding a URL with an invalid protocol to a url claim.""" repo = self.get_repo() item = pywikibot.ItemPage(repo, 'Q68') claim = pywikibot.page.Claim(repo, 'P506', datatype='url') -- To view, visit https://gerrit.wikimedia.org/r/347333 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ida681e2a71d261ef3afcf34387a29388655856a7 Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Lokal Profil <lokal.profil(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: Add support for geo-shape Wikibase data type

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/346700 ) Change subject: Add support for geo-shape Wikibase data type ...................................................................... Add support for geo-shape Wikibase data type Introduces the WbGeoShape class to provide handling of geo-shape data and do some basic validation on indata. Also add a comment to commonsMedia to the task explaining how come site is hard-coded to Wikimedia Commons for all Wikibase installations (which is not the case for geo-shapes). Bug:T161726 Change-Id: Ia96d35519163307227159ccece5981783ce8b558 --- M pywikibot/__init__.py M pywikibot/families/wikidata_family.py M pywikibot/page.py M pywikibot/site.py M tests/wikibase_edit_tests.py M tests/wikibase_tests.py 6 files changed, 171 insertions(+), 3 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py index eff852a..1b6ce51 100644 --- a/pywikibot/__init__.py +++ b/pywikibot/__init__.py @@ -812,6 +812,73 @@ return cls(wb['text'], wb['language']) +class WbGeoShape(_WbRepresentation): + """ + A Wikibase geo-shape representation. + + A temporary implementation until T162336 has been resolved. + """ + + _items = ('page', ) + + def __init__(self, page, site=None): + """ + Create a new WbGeoShape object. + + @param page: page containing the map data + @type text: pywikibot.Page + @param site: The Wikibase site + @type site: pywikibot.site.DataSite + """ + site = site or Site().data_repository() + if not isinstance(page, Page): + raise ValueError('page must be a pywikibot.Page object.') + + # validate page exists + if not page.exists(): + raise ValueError('page must exist.') + + # validate page is on the right site, and that site supports geo-shapes + geo_shape_site = site.geo_shape_repository() + if not geo_shape_site: + raise ValueError('the provided site does not support geo-shapes.') + if page.site != geo_shape_site: + raise ValueError('page must be on the image repository site.') + + # validate page title fulfills hard-coded Wikibase requirement + # pcre regexp: '/^Data:[^\\[\\]#\\\:{|}]+\.map$/u' + # As we have already checked for existence the following simplified + # check should be enough. + if not page.title().startswith('Data:') or \ + not page.title().endswith('.map'): + raise ValueError( + "page must be a '.map' page in the 'Data:' namespace.") + + self.page = page + + def toWikibase(self): + """ + Convert the data to the value required by the Wikibase API. + + @return: title of the geo-shape page incl. namespace + @rtype: str + """ + return self.page.title() + + @classmethod + def fromWikibase(cls, page_name, site): + """ + Create a WbGeoShape from the JSON data given by the Wikibase API. + + @param page_name: page name from Wikibase value + @type page_name: str + @rtype: pywikibot.WbGeoShape + """ + geo_shape_site = site.geo_shape_repository() + page = Page(geo_shape_site, page_name) + return cls(page, site) + + _sites = {} _url_cache = {} # The code/fam pair for each URL diff --git a/pywikibot/families/wikidata_family.py b/pywikibot/families/wikidata_family.py index f695a20..4b30106 100644 --- a/pywikibot/families/wikidata_family.py +++ b/pywikibot/families/wikidata_family.py @@ -50,6 +50,11 @@ """Default calendar model for WbTime datatype.""" return 'http://www.wikidata.org/entity/Q1985727' + def shared_geo_shape_repository(self, code): + """Return Wikimedia Commons as the repository for geo-shapes.""" + # Per geoShapeStorageFrontendUrl settings in Wikibase + return ('commons', 'commons') + def globes(self, code): """Supported globes for Coordinate datatype.""" return { diff --git a/pywikibot/page.py b/pywikibot/page.py index 1885aef..e95bab3 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -4318,6 +4318,7 @@ 'monolingualtext': pywikibot.WbMonolingualText, 'math': basestring, 'external-id': basestring, + 'geo-shape': pywikibot.WbGeoShape, } value_types = {'wikibase-item': 'wikibase-entityid', @@ -4327,6 +4328,7 @@ 'globe-coordinate': 'globecoordinate', 'math': 'string', 'external-id': 'string', + 'geo-shape': 'string', } def __init__(self, site, id, datatype=None): @@ -4457,8 +4459,9 @@ 'wikibase-property': lambda value, site: PropertyPage(site, 'P' + str(value['numeric-id'])), 'commonsMedia': lambda value, site: - FilePage(pywikibot.Site('commons', 'commons'), value), + FilePage(pywikibot.Site('commons', 'commons'), value), # T90492 'globe-coordinate': pywikibot.Coordinate.fromWikibase, + 'geo-shape': pywikibot.WbGeoShape.fromWikibase, 'time': lambda value, site: pywikibot.WbTime.fromWikibase(value), 'quantity': pywikibot.WbQuantity.fromWikibase, 'monolingualtext': lambda value, site: @@ -4884,7 +4887,8 @@ elif self.type == 'commonsMedia': value = self.getTarget().title(withNamespace=False) elif self.type in ('globe-coordinate', 'time', - 'quantity', 'monolingualtext'): + 'quantity', 'monolingualtext', + 'geo-shape'): value = self.getTarget().toWikibase() else: raise NotImplementedError('%s datatype is not supported yet.' diff --git a/pywikibot/site.py b/pywikibot/site.py index c715902..7c04551 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -7231,6 +7231,13 @@ """ return self + def geo_shape_repository(self): + """Return Site object for the geo-shapes repository e.g. commons.""" + # Do this via API instead if T162561 is implemented. + code, fam = self.shared_geo_shape_repository() + if bool(code or fam): + return pywikibot.Site(code, fam, self.username()) + def loadcontent(self, identification, *props): """ Fetch the current content of a Wikibase item. diff --git a/tests/wikibase_edit_tests.py b/tests/wikibase_edit_tests.py index 670eab9..1361c98 100644 --- a/tests/wikibase_edit_tests.py +++ b/tests/wikibase_edit_tests.py @@ -300,6 +300,25 @@ claim = item.claims['P718'][0] self.assertEqual(claim.getTarget(), target) + def test_WbGeoShape_edit(self): + """Attempt adding a geo-shape with valid input.""" + # Clean the slate in preparation for test. + testsite = self.get_repo() + item = self._clean_item(testsite, 'P27199') + + # set new claim + claim = pywikibot.page.Claim(testsite, 'P27199', datatype='geo-shape') + commons_site = pywikibot.Site('commons', 'commons') + page = pywikibot.Page(commons_site, 'Data:Lyngby Hovedgade.map') + target = pywikibot.WbGeoShape(page) + claim.setTarget(target) + item.addClaim(claim) + + # confirm new claim + item.get(force=True) + claim = item.claims['P27199'][0] + self.assertEqual(claim.getTarget(), target) + class TestWikibaseRemoveQualifier(WikibaseTestCase): diff --git a/tests/wikibase_tests.py b/tests/wikibase_tests.py index 0006172..98c542d 100644 --- a/tests/wikibase_tests.py +++ b/tests/wikibase_tests.py @@ -18,7 +18,7 @@ import pywikibot from pywikibot import pagegenerators -from pywikibot.page import WikibasePage, ItemPage, PropertyPage +from pywikibot.page import WikibasePage, ItemPage, PropertyPage, Page from pywikibot.site import Namespace, NamespacesDict from pywikibot.tools import MediaWikiVersion @@ -490,6 +490,72 @@ text=None, language='sv') +class TestWbGeoShapeNonDry(WikidataTestCase): + + """ + Test Wikibase WbGeoShape data type (non-dry). + + These require non dry tests due to the page.exists() call. + """ + + def setUp(self): + """Setup tests.""" + self.commons = pywikibot.Site('commons', 'commons') + self.page = Page(self.commons, 'Data:Lyngby Hovedgade.map') + super(TestWbGeoShapeNonDry, self).setUp() + + def test_WbGeoShape_page(self): + """Test WbGeoShape page.""" + q = pywikibot.WbGeoShape(self.page) + q_val = u'Data:Lyngby Hovedgade.map' + self.assertEqual(q.toWikibase(), q_val) + + def test_WbGeoShape_page_and_site(self): + """Test WbGeoShape from page and site.""" + q = pywikibot.WbGeoShape(self.page, self.get_repo()) + q_val = u'Data:Lyngby Hovedgade.map' + self.assertEqual(q.toWikibase(), q_val) + + def test_WbGeoShape_equality(self): + """Test WbGeoShape equality.""" + q = pywikibot.WbGeoShape(self.page, self.get_repo()) + self.assertEqual(q, q) + + def test_WbGeoShape_fromWikibase(self): + """Test WbGeoShape.fromWikibase() instantiating.""" + repo = self.get_repo() + q = pywikibot.WbGeoShape.fromWikibase( + 'Data:Lyngby Hovedgade.map', repo) + self.assertEqual(q.toWikibase(), 'Data:Lyngby Hovedgade.map') + + def test_WbGeoShape_error_on_non_page(self): + """Test WbGeoShape error handling when given a non-page.""" + self.assertRaises(ValueError, pywikibot.WbGeoShape, + 'A string', self.get_repo()) + + def test_WbGeoShape_error_on_non_exitant_page(self): + """Test WbGeoShape error handling of a non-existant page.""" + page = Page(self.commons, 'Non-existant page... really') + self.assertRaises(ValueError, pywikibot.WbGeoShape, + page, self.get_repo()) + + def test_WbGeoShape_error_on_wrong_site(self): + """Test WbGeoShape error handling of a page on non-filerepo site.""" + repo = self.get_repo() + page = Page(repo, 'Q123') + self.assertRaises(ValueError, pywikibot.WbGeoShape, + page, self.get_repo()) + + def test_WbGeoShape_error_on_wrong_page_type(self): + """Test WbGeoShape error handling of a non-map page.""" + non_data_page = Page(self.commons, 'File:Foo.jpg') + non_map_page = Page(self.commons, 'Data:Templatedata/Graph:Lines.tab') + self.assertRaises(ValueError, pywikibot.WbGeoShape, + non_data_page, self.get_repo()) + self.assertRaises(ValueError, pywikibot.WbGeoShape, + non_map_page, self.get_repo()) + + class TestItemPageExtensibility(TestCase): """Test ItemPage extensibility.""" -- To view, visit https://gerrit.wikimedia.org/r/346700 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia96d35519163307227159ccece5981783ce8b558 Gerrit-PatchSet: 6 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Lokal Profil <lokal.profil(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Lokal Profil <lokal.profil(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: [bugfix] Re-enable getFileVersionHistoryTable() method

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/347233 ) Change subject: [bugfix] Re-enable getFileVersionHistoryTable() method ...................................................................... [bugfix] Re-enable getFileVersionHistoryTable() method - getFileVersionHistoryTable always failed due to missing joined string for join statement - re-enable this method - use new translatable table header used by commons - use size inside dimension field Bug: T162528 Change-Id: I96b9ebd72496cd73bd2710dce9599c85fc346c66 --- M pywikibot/page.py 1 file changed, 8 insertions(+), 9 deletions(-) Approvals: XXN: Looks good to me, but someone else must approve Dalba: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index 00c60c3..a5bf566 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -2462,15 +2462,14 @@ """Return the version history in the form of a wiki table.""" lines = [] for info in self.getFileVersionHistory(): - datetime = info['timestamp'] - username = info['user'] - resolution = '%dx%d' % (info['height'], info['width']) - size = info['size'] - comment = info['comment'] - lines.append(u'| %s || %s || %s || %s || <nowiki>%s</nowiki>' - % (datetime, username, resolution, size, comment)) - return ('{| border="1"\n! date/time || username || resolution || size ' - '|| edit summary\n|----\n\n|----\n'.join(lines) + '\n|}') + dimension = '{width}×{height} px ({size} bytes)'.format(**info) + lines.append('| {timestamp} || {user} || {dimension} |' + '| <nowiki>{comment}</nowiki>' + ''.format(dimension=dimension, **info)) + return ('{| class="wikitable"\n' + '! {{int:filehist-datetime}} || {{int:filehist-user}} |' + '| {{int:filehist-dimensions}} || {{int:filehist-comment}}\n' + '|-\n%s\n|}\n' % '\n|-\n'.join(lines)) @deprecated_args(step=None) def usingPages(self, total=None, content=False): -- To view, visit https://gerrit.wikimedia.org/r/347233 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I96b9ebd72496cd73bd2710dce9599c85fc346c66 Gerrit-PatchSet: 2 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: XXN <dan15i(a)yahoo.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: Add async parameter to ItemPage.editEntity

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/258684 ) Change subject: Add async parameter to ItemPage.editEntity ...................................................................... Add async parameter to ItemPage.editEntity 'asynchronous' parameter enables ItemPage.editEntity to edit an entity asynchronously, which can be used in Wikidata bots. TODO: - needs better exception handling? (T86083) - new callback argument, just like in BasePage.save() Bug: T86074 Change-Id: I1ffa92b8710e91b7095b24808c151ff9d0a592cf --- M pywikibot/bot.py M pywikibot/page.py 2 files changed, 8 insertions(+), 2 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/bot.py b/pywikibot/bot.py index 80905bf..9a24644 100644 --- a/pywikibot/bot.py +++ b/pywikibot/bot.py @@ -1866,7 +1866,6 @@ if 'summary' in kwargs: pywikibot.output(u'Change summary: %s' % kwargs['summary']) - # TODO async in editEntity should actually have some effect (bug T86074) # TODO PageSaveRelatedErrors should be actually raised in editEntity # (bug T86083) self._save_page(item, item.editEntity, data, **kwargs) diff --git a/pywikibot/page.py b/pywikibot/page.py index dc34774..a1540f4 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -3541,7 +3541,7 @@ return site.dbName() return site - def editEntity(self, data=None, **kwargs): + def editEntity(self, data=None, asynchronous=False, **kwargs): """ Edit an entity using Wikibase wbeditentity API. @@ -3553,7 +3553,14 @@ @param data: Data to be saved @type data: dict, or None to save the current content of the entity. + @param asynchronous: if True, launch a separate thread to edit + asynchronously + @type asynchronous: bool """ + if asynchronous: + pywikibot.async_request(self.editEntity, data, **kwargs) + return + if hasattr(self, '_revid'): baserevid = self.latest_revision_id else: -- To view, visit https://gerrit.wikimedia.org/r/258684 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I1ffa92b8710e91b7095b24808c151ff9d0a592cf Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Vadiraja.k <vadi.fedx(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <Ladsgroup(a)gmail.com> Gerrit-Reviewer: M4tx <m4tx(a)m4tx.pl> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: MarcoAurelio <strigiwm(a)gmail.com> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: [IMPR] Polish WikidataBot.user_edit_entity

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/332333 ) Change subject: [IMPR] Polish WikidataBot.user_edit_entity ...................................................................... [IMPR] Polish WikidataBot.user_edit_entity - Fix accessing WikibasePage. - Make the method return bool, just like BaseBot._save_page does. - Do not automatically set current_item, this shows ugly "-1" for new items. Bug: T87493 Change-Id: I91bf76085ce19bf191dcdbab73b2fb7c1b5bfac4 --- M pywikibot/bot.py 1 file changed, 5 insertions(+), 5 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/bot.py b/pywikibot/bot.py index 82f24ef..f35a785 100644 --- a/pywikibot/bot.py +++ b/pywikibot/bot.py @@ -1313,7 +1313,7 @@ @rtype: bool """ if not self.user_confirm('Do you want to accept these changes?'): - return + return False if 'async' not in kwargs and self.getOption('always'): kwargs['async'] = True @@ -1871,16 +1871,16 @@ @kwarg ignore_save_related_errors: if True, errors related to page save will be reported and ignored (default: False) @type ignore_save_related_errors: bool + @return: whether the item was saved successfully + @rtype: bool """ - self.current_page = item - show_diff = kwargs.pop('show_diff', True) if show_diff: if data is None: diff = item.toJSON(diffto=( item._content if hasattr(item, '_content') else None)) else: - diff = pywikibot.WikibasePage._normalizeData(data) + diff = pywikibot.page.WikibasePage._normalizeData(data) pywikibot.output(json.dumps(diff, indent=4, sort_keys=True)) if 'summary' in kwargs: @@ -1889,7 +1889,7 @@ # TODO async in editEntity should actually have some effect (bug T86074) # TODO PageSaveRelatedErrors should be actually raised in editEntity # (bug T86083) - self._save_page(item, item.editEntity, data, **kwargs) + return self._save_page(item, item.editEntity, data, **kwargs) def getSource(self, site): """ -- To view, visit https://gerrit.wikimedia.org/r/332333 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I91bf76085ce19bf191dcdbab73b2fb7c1b5bfac4 Gerrit-PatchSet: 9 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: Make sparql use Site to access sparql endpoint and entity_url

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/319857 ) Change subject: Make sparql use Site to access sparql endpoint and entity_url ...................................................................... Make sparql use Site to access sparql endpoint and entity_url Introduces a repo parameter which takes any DataSite and gets its sparql endpoint and entity_url. Still supports explicit endpoint and entity_url for full flexibility. As before this defaults to Wikidata but now relies on its Site object rather than hard-coded endpoint and entity_url. The same change is implemented in pagegenerators. Note that this change is breaking for anyone who had overridden endpoint but not entity_url (i.e. running their own endpoint but loaded with wikidata data). This was an undocumented feature and these users will now recieve an Error. Bug: T159956 Change-Id: I057335a56398a7d5f6c8c811233a002228b73093 --- M pywikibot/data/sparql.py M pywikibot/pagegenerators.py M pywikibot/site.py M tests/sparql_tests.py 4 files changed, 64 insertions(+), 15 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/data/sparql.py b/pywikibot/data/sparql.py index e95c916..b577bb0 100644 --- a/pywikibot/data/sparql.py +++ b/pywikibot/data/sparql.py @@ -1,6 +1,8 @@ # -*- coding: utf-8 -*- """SPARQL Query interface.""" # +# (C) Pywikibot team, 2016-2017 +# # Distributed under the terms of the MIT license. # from __future__ import absolute_import, unicode_literals @@ -12,10 +14,10 @@ else: from urllib2 import quote +from pywikibot import Site, Error from pywikibot.comms import http from pywikibot.tools import UnicodeMixin, py2_encode_utf_8 -WIKIDATA = 'http://query.wikidata.org/sparql' DEFAULT_HEADERS = {'cache-control': 'no-cache', 'Accept': 'application/sparql-results+json'} @@ -27,15 +29,44 @@ This class allows to run SPARQL queries against any SPARQL endpoint. """ - def __init__(self, endpoint=WIKIDATA, entity_url='http://www.wikidata.org/entity/'): + def __init__(self, endpoint=None, entity_url=None, repo=None): """ Create endpoint. - @param endpoint: SPARQL endpoint URL, by default Wikidata query endpoint + @param endpoint: SPARQL endpoint URL + @type endpoint: string + @param entity_url: URL prefix for any entities returned in a query. + @type entity_url: string + @param repo: The Wikibase site which we want to run queries on. If + provided this overrides any value in endpoint and entity_url. + Defaults to Wikidata. + @type repo: pywikibot.site.DataSite """ - self.endpoint = endpoint + # default to Wikidata + if not repo and not endpoint: + repo = Site('wikidata', 'wikidata') + + if repo: + try: + self.endpoint = repo.sparql_endpoint + self.entity_url = repo.concept_base_uri + except NotImplementedError: + raise NotImplementedError( + 'Wiki version must be 1.28-wmf.23 or newer to ' + 'automatically extract the sparql endpoint. ' + 'Please provide the endpoint and entity_url ' + 'parameters instead of a repo.') + if not self.endpoint: + raise Error('The site {0} does not provide a sparql endpoint.' + .format(repo)) + else: + if not entity_url: + raise Error('If initialised with an endpoint the entity_url ' + 'must be provided.') + self.endpoint = endpoint + self.entity_url = entity_url + self.last_response = None - self.entity_url = entity_url def get_last_response(self): """ diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index 61dacdb..1152def 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -40,13 +40,13 @@ from pywikibot.tools import ( deprecated, deprecated_args, - redirect_func, - issue_deprecation_warning, - itergroup, DequeGenerator, - intersect_generators, - IteratorNextMixin, filter_unique, + intersect_generators, + issue_deprecation_warning, + IteratorNextMixin, + itergroup, + redirect_func, ) from pywikibot import date, config, i18n, xmlreader @@ -2727,7 +2727,16 @@ site = pywikibot.Site() repo = site.data_repository() if endpoint is None: - endpoint = sparql.WIKIDATA + try: + endpoint = repo.sparql_endpoint + except NotImplementedError: + raise NotImplementedError( + 'Wiki version must be 1.28-wmf.23 or newer to automatically ' + 'extract the sparql endpoint. Please provide the endpoint ' + 'parameter as well.') + if not endpoint: + pywikibot.error('The site {0} does not provide a sparql endpoint.' + .format(repo)) query_object = sparql.SparqlQuery(endpoint=endpoint) data = query_object.get_items(query, diff --git a/pywikibot/site.py b/pywikibot/site.py index f2724d9..5da6345 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -7057,6 +7057,17 @@ % self) @property + @need_version("1.28-wmf.3") + def sparql_endpoint(self): + """ + Return the sparql endpoint url, if any has been set. + + @return: sparql endpoint url + @rtype: str|None + """ + return self.siteinfo['general'].get('wikibase-sparql') + + @property @need_version("1.28-wmf.23") def concept_base_uri(self): """ diff --git a/tests/sparql_tests.py b/tests/sparql_tests.py index 02c76d6..c392d57 100644 --- a/tests/sparql_tests.py +++ b/tests/sparql_tests.py @@ -12,7 +12,7 @@ import pywikibot.data.sparql as sparql from pywikibot.tools import UnicodeType -from tests.aspects import unittest, TestCase +from tests.aspects import unittest, TestCase, WikidataTestCase if sys.version_info[0] > 2: from unittest.mock import patch @@ -95,10 +95,8 @@ self.content = value -class TestSparql(TestCase): +class TestSparql(WikidataTestCase): """Test SPARQL queries.""" - - net = False @patch.object(sparql.http, 'fetch') def testQuerySelect(self, mock_method): -- To view, visit https://gerrit.wikimedia.org/r/319857 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I057335a56398a7d5f6c8c811233a002228b73093 Gerrit-PatchSet: 11 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Lokal Profil <lokal.profil(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <Ladsgroup(a)gmail.com> Gerrit-Reviewer: Lokal Profil <lokal.profil(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Smalyshev <smalyshev(a)wikimedia.org> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: [Bug] Create attributes when item is being preloaded

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/338631 ) Change subject: [Bug] Create attributes when item is being preloaded ...................................................................... [Bug] Create attributes when item is being preloaded Attributes like labels or claims are only created inside ItemPage.get(). Change-Id: Iec24389546725dd8142374ee755fedea7284fee3 --- M pywikibot/site.py 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/site.py b/pywikibot/site.py index bdf1fb4..7c453fa 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -7231,6 +7231,8 @@ for qid in data['entities']: item = pywikibot.ItemPage(self, qid) item._content = data['entities'][qid] + # No api call is made because item._content is given + item.get(get_redirect=True) yield item def getPropertyType(self, prop): -- To view, visit https://gerrit.wikimedia.org/r/338631 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iec24389546725dd8142374ee755fedea7284fee3 Gerrit-PatchSet: 4 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Lokal Profil <lokal.profil(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

[Gerrit] pywikibot/core[master]: timestripper: search wikilinks to reduce false matches

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/337293 ) Change subject: timestripper: search wikilinks to reduce false matches ...................................................................... timestripper: search wikilinks to reduce false matches Search timestamp in wikilinks as well. After searching, wikilinks are censored, so possible digits in wikilinks will not cause false matches, while allowing to maintain gaps (used in _valid_date_dict_positions()). Hyperlinks are censored as well, to avoid false matches. Deprecated: - Timestripper.linkP, in favour of an internal variable. - Timestripper.comment_pattern, in favour of an internal variable. Added tests for comments, wikilinks and hyperlinks. Change-Id: I6fb15b2eed6845e57b31042f9312890025b1b7c7 --- M pywikibot/textlib.py M tests/timestripper_tests.py 2 files changed, 154 insertions(+), 5 deletions(-) Approvals: jenkins-bot: Verified Xqt: Looks good to me, approved diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py index 57c338a..75a68c6 100644 --- a/pywikibot/textlib.py +++ b/pywikibot/textlib.py @@ -7,7 +7,7 @@ """ # -# (C) Pywikibot team, 2008-2016 +# (C) Pywikibot team, 2008-2017 # # Distributed under the terms of the MIT license. # @@ -1927,11 +1927,25 @@ self.pdayR, ] - self.linkP = compileLinkR() - self.comment_pattern = re.compile(r'') + self._hyperlink_pat = re.compile(r'\[\s*?http[s]?://[^\]]*?\]') + self._comment_pat = re.compile(r'') + self._wikilink_pat = re.compile( + r'\[\[(?P<link>[^\]\|]*?)(?P<anchor>\|[^\]]*)?\]\]') self.tzinfo = tzoneFixedOffset(self.site.siteinfo['timeoffset'], self.site.siteinfo['timezone']) + + @property + @deprecated('_hyperlink_pat') + def linkP(self): + """Deprecated linkP instance variable.""" + return self._hyperlink_pat + + @property + @deprecated('_comment_pat') + def comment_pattern(self): + """Deprecated comment_pattern instance variable.""" + return self._comment_pat @deprecated('module function') def findmarker(self, text, base=u'@@', delta='@'): @@ -2014,23 +2028,47 @@ @return: A timestamp found on the given line @rtype: pywikibot.Timestamp """ + # Try to maintain gaps that are used in _valid_date_dict_positions() + def censor_match(match): + return '_' * (match.end() - match.start()) + # match date fields dateDict = dict() + # Analyze comments separately from rest of each line to avoid to skip # dates in comments, as the date matched by timestripper is the # rightmost one. most_recent = [] - for comment in self.comment_pattern.finditer(line): + for comment in self._comment_pat.finditer(line): # Recursion levels can be maximum two. If a comment is found, it will # not for sure be found in the next level. # Nested comments are excluded by design. timestamp = self.timestripper(comment.group(1)) most_recent.append(timestamp) + # Censor comments. + line = self._comment_pat.sub(censor_match, line) + + # Censor external links. + line = self._hyperlink_pat.sub(censor_match, line) + + for wikilink in self._wikilink_pat.finditer(line): + # Recursion levels can be maximum two. If a link is found, it will + # not for sure be found in the next level. + # Nested links are excluded by design. + link, anchor = wikilink.group('link'), wikilink.group('anchor') + timestamp = self.timestripper(link) + most_recent.append(timestamp) + if anchor: + timestamp = self.timestripper(anchor) + most_recent.append(timestamp) + + # Censor wikilinks. + line = self._wikilink_pat.sub(censor_match, line) + # Remove parts that are not supposed to contain the timestamp, in order # to reduce false positives. line = removeDisabledParts(line) - line = self.linkP.sub('', line) # remove external links line = self.fix_digits(line) for pat in self.patterns: diff --git a/tests/timestripper_tests.py b/tests/timestripper_tests.py index 125f7c1..ec2d0b5 100644 --- a/tests/timestripper_tests.py +++ b/tests/timestripper_tests.py @@ -272,6 +272,117 @@ self.assertEqual(self.ts.timestripper(txtNoMatch), None) +class TestTimeStripperTreatSpecialText(TestTimeStripperCase): + + """Test special text behaviour (comments, hyperlinks, wikilinks).""" + + family = 'wikisource' + code = 'en' + + date = '06:57 06 June 2015 (UTC)' + fake_date = '05:57 06 June 2015 (UTC)' + tzone = tzoneFixedOffset(0, 'UTC') + expected_date = datetime.datetime(2015, 6, 6, 6, 57, tzinfo=tzone) + + def test_timestripper_match_comment(self): + """Test that comments are correctly matched.""" + ts = self.ts + + txt_match = self.date + '' + exp_match = 'a test comment' + self.assertEqual(ts._comment_pat.search(txt_match).group(1), + exp_match) + + def test_timestripper_match_hyperlink(self): + """Test that hyperlinks are correctly matched.""" + ts = self.ts + + txt_match = '[http://test.org | a link]' + exp_match = '[http://test.org | a link]' + self.assertEqual(ts._hyperlink_pat.search(txt_match).group(), + exp_match) + + def test_timestripper_match_wikilink(self): + """Test that wikilinks are correctly matched.""" + ts = self.ts + + txt_match = '[[wikilink|a wikilink with no date]]' + exp_match_link = 'wikilink' + exp_match_anchor = '|a wikilink with no date' + self.assertEqual(ts._wikilink_pat.search(txt_match).group('link'), + exp_match_link) + self.assertEqual(ts._wikilink_pat.search(txt_match).group('anchor'), + exp_match_anchor) + + def test_timestripper_match_comment_with_date(self): + """Test that dates in comments are correctly matched.""" + ts = self.ts.timestripper + + txt_match = self.date + '' + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = '' + self.date + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = '' + self.fake_date + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = '' + self.fake_date + self.assertEqual(ts(txt_match), self.expected_date) + + def test_timestripper_skip_hyperlink(self): + """Test that dates in hyperlinks are correctly skipped.""" + ts = self.ts.timestripper + + txt_match = self.date + '[http://' + self.fake_date + ']' + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = '[http://' + self.fake_date + ']' + self.date + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = ('%s [http://www.org | link with date %s]' + % (self.date, self.fake_date)) + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = '[http://' + self.fake_date + ']' + self.date + self.assertEqual(ts(txt_match), self.expected_date) + + def test_timestripper_skip_hyperlink_and_do_not_connect(self): + """Test that skipping hyperlinks will not make gaps shorter.""" + ts = self.ts.timestripper + + txt_match = ('%s[http://example.com Here is long enough text]%s' + % (self.date[:9], self.date[9:])) + self.assertEqual(ts(txt_match), None) + + def test_timestripper_match_wikilink_with_date(self): + """Test that dates in wikilinks are correctly matched.""" + ts = self.ts.timestripper + + txt_match = self.date + '[[' + self.fake_date + ']]' + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = '[[' + self.fake_date + ']]' + self.date + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = '[[' + self.date + ']]' + self.fake_date + self.assertEqual(ts(txt_match), self.expected_date) + + txt_match = '[[wikilink|' + self.date + ']]' + self.fake_date + self.assertEqual(ts(txt_match), self.expected_date) + + def test_timestripper_skip_wikilink_and_do_not_connect(self): + """Test that skipping wikilinks will not make gaps shorter.""" + ts = self.ts.timestripper + + txt_match = ('%s[[Here is long enough text]]%s' + % (self.date[:9], self.date[9:])) + self.assertEqual(ts(txt_match), None) + + txt_match = self.date[:9] + '[[foo]]' + self.date[9:] + self.assertEqual(ts(txt_match), self.expected_date) + + class TestTimeStripperDoNotArchiveUntil(TestTimeStripperCase): """Test cases for Do Not Archive Until templates. -- To view, visit https://gerrit.wikimedia.org/r/337293 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I6fb15b2eed6845e57b31042f9312890025b1b7c7 Gerrit-PatchSet: 18 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Whym <whym(a)whym.org> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

7 years

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits April 2017