Pywikibot-commits August 2014

pywikibot-commits@lists.wikimedia.org

1 participants
196 discussions

[Gerrit] Ensure genFactory.namespaces is always ints - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Ensure genFactory.namespaces is always ints ...................................................................... Ensure genFactory.namespaces is always ints Change-Id: Ibcf673119f755cd7003045c269a2c382cd5b92c7 --- M pywikibot/pagegenerators.py 1 file changed, 27 insertions(+), 25 deletions(-) Approvals: John Vandenberg: Looks good to me, but someone else must approve Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index aad6df1..39b38b9 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -70,9 +70,9 @@ -search Work on all pages that are found in a MediaWiki search across all namespaces. --namespace Filter the page generator to only yield pages in the --ns specified namespaces. Separate multiple namespace - numbers with commas. Example "-ns:0,2,4" +-namespaces Filter the page generator to only yield pages in the +-namespace specified namespaces. Separate multiple namespace +-ns numbers with commas. Example "-ns:0,2,4" If used with -newpages, -namepace/ns must be provided before -newpages. If used with -recentchanges, efficiency is improved if @@ -220,11 +220,10 @@ if gen: self.gens.insert(0, gen) - namespaces = [int(n) for n in self.namespaces] for i in range(len(self.gens)): if isinstance(self.gens[i], pywikibot.data.api.QueryGenerator): if self.namespaces: - self.gens[i].set_namespace(namespaces) + self.gens[i].set_namespace(self.namespaces) if self.step: self.gens[i].set_query_increment(self.step) if self.limit: @@ -232,7 +231,7 @@ else: if self.namespaces: self.gens[i] = NamespaceFilterPageGenerator(self.gens[i], - namespaces) + self.namespaces) if self.limit: self.gens[i] = itertools.islice(self.gens[i], self.limit) if len(self.gens) == 0: @@ -352,11 +351,10 @@ else: gen = RandomPageGenerator(total=int(arg[8:])) elif arg.startswith('-recentchanges'): - namespaces = [int(n) for n in self.namespaces] or None if len(arg) >= 15: - gen = RecentChangesPageGenerator(namespaces=namespaces, total=int(arg[15:])) + gen = RecentChangesPageGenerator(namespaces=self.namespaces, total=int(arg[15:])) else: - gen = RecentChangesPageGenerator(namespaces=namespaces, total=60) + gen = RecentChangesPageGenerator(namespaces=self.namespaces, total=60) gen = DuplicateFilterPageGenerator(gen) elif arg.startswith('-file'): textfilename = arg[6:] @@ -364,19 +362,24 @@ textfilename = pywikibot.input( u'Please enter the local file name:') gen = TextfilePageGenerator(textfilename) - elif arg.startswith('-namespace'): - if len(arg) == len('-namespace'): - self.namespaces.append( - pywikibot.input(u'What namespace are you filtering on?')) - else: - self.namespaces.extend(arg[len('-namespace:'):].split(",")) - return True - elif arg.startswith('-ns'): - if len(arg) == len('-ns'): - self.namespaces.append( - pywikibot.input(u'What namespace are you filtering on?')) - else: - self.namespaces.extend(arg[len('-ns:'):].split(",")) + elif arg.startswith('-namespace') or arg.startswith('-ns'): + value = None + if arg.startswith('-ns:'): + value = arg[len('-ns:'):] + elif arg.startswith('-namespace:'): + value = arg[len('-namespace:'):] + elif arg.startswith('-namespaces:'): + value = arg[len('-namespaces:'):] + if not value: + value = pywikibot.input( + u'What namespace are you filtering on?') + try: + self.namespaces.extend( + [int(ns) for ns in value.split(",")] + ) + except ValueError: + pywikibot.output(u'Invalid namespaces argument: %s' % value) + return False return True elif arg.startswith('-step'): if len(arg) == len('-step'): @@ -477,11 +480,10 @@ # partial workaround for bug 67249 # to use -namespace/ns with -newpages, -ns must be given before -newpages # otherwise default namespace is 0 - namespaces = [int(n) for n in self.namespaces] or 0 if len(arg) >= 10: - gen = NewpagesPageGenerator(namespaces=namespaces, total=int(arg[10:])) + gen = NewpagesPageGenerator(namespaces=self.namespaces, total=int(arg[10:])) else: - gen = NewpagesPageGenerator(namespaces=namespaces, total=60) + gen = NewpagesPageGenerator(namespaces=self.namespaces, total=60) elif arg.startswith('-imagesused'): imagelinkstitle = arg[len('-imagesused:'):] if not imagelinkstitle: -- To view, visit https://gerrit.wikimedia.org/r/151816 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ibcf673119f755cd7003045c269a2c382cd5b92c7 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] rename ImagePage to FilePage - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: rename ImagePage to FilePage ...................................................................... rename ImagePage to FilePage for consistency with the MediaWiki canonical namespace name; also, ImagePage did not actually expose any methods peculiar to images - replaced all occurrences - added a notice to README-conversion.txt - updated compat2core.py to make replacements easier Change-Id: I289da42fa4b5f78e4837c8e4c1a3a72e9d322990 --- M README-conversion.txt M pywikibot/__init__.py M pywikibot/data/api.py M pywikibot/page.py M pywikibot/pagegenerators.py M pywikibot/site.py M scripts/checkimages.py M scripts/harvest_template.py M scripts/illustrate_wikidata.py M scripts/image.py M scripts/imagerecat.py M scripts/imagetransfer.py M scripts/maintenance/compat2core.py M scripts/nowcommons.py M scripts/upload.py M tests/page_tests.py M tests/site_tests.py 17 files changed, 119 insertions(+), 115 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/README-conversion.txt b/README-conversion.txt index b344c33..b137a28 100644 --- a/README-conversion.txt +++ b/README-conversion.txt @@ -73,7 +73,7 @@ this below) handles link parsing and interpretation that doesn't require access to the wiki server. -A third syntax allows easy conversion from a Page object to an ImagePage or +A third syntax allows easy conversion from a Page object to a FilePage or Category, or vice versa: e.g., Category(pageobj) converts a Page to a Category, as long as the page is in the category namespace. @@ -99,9 +99,10 @@ - getVersionHistory(): Returns a pywikibot.Timestamp object instead of a MediaWiki one -=== ImagePage objects === +=== FilePage objects === -For ImagePage objects, the getFileMd5Sum() method is deprecated; it is +The old ImagePage class has been renamed into FilePage. +For FilePage objects, the getFileMd5Sum() method is deprecated; it is recommended to replace it with getFileSHA1Sum(), because MediaWiki now stores the SHA1 hash of images. diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py index e29691f..b3ff554 100644 --- a/pywikibot/__init__.py +++ b/pywikibot/__init__.py @@ -55,7 +55,7 @@ __all__ = ( 'config', 'ui', 'UnicodeMixin', 'translate', - 'Page', 'ImagePage', 'Category', 'Link', 'User', + 'Page', 'FilePage', 'ImagePage', 'Category', 'Link', 'User', 'ItemPage', 'PropertyPage', 'Claim', 'TimeStripper', 'html2unicode', 'url2unicode', 'unicode2html', 'stdout', 'output', 'warning', 'error', 'critical', 'debug', 'exception', @@ -533,7 +533,7 @@ getSite = Site # alias for backwards-compability -from .page import Page, ImagePage, Category, Link, User, ItemPage, PropertyPage, Claim +from .page import Page, FilePage, ImagePage, Category, Link, User, ItemPage, PropertyPage, Claim from .page import html2unicode, url2unicode, unicode2html diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py index 9ff7ba7..b02c263 100644 --- a/pywikibot/data/api.py +++ b/pywikibot/data/api.py @@ -927,14 +927,14 @@ class ImagePageGenerator(PageGenerator): - """Like PageGenerator, but yields ImagePage objects instead of Pages.""" + """Like PageGenerator, but yields FilePage objects instead of Pages.""" def result(self, pagedata): p = PageGenerator.result(self, pagedata) - image = pywikibot.ImagePage(p) + filepage = pywikibot.FilePage(p) if 'imageinfo' in pagedata: - image._imageinfo = pagedata['imageinfo'][0] - return image + filepage._imageinfo = pagedata['imageinfo'][0] + return filepage class PropertyGenerator(QueryGenerator): diff --git a/pywikibot/page.py b/pywikibot/page.py index 7108a2b..fb5bffe 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -71,7 +71,7 @@ - If the first argument is a Page, create a copy of that object. This can be used to convert an existing Page into a subclass - object, such as Category or ImagePage. (If the title is also + object, such as Category or FilePage. (If the title is also given as the second argument, creates a copy with that title; this is used when pages are moved.) - If the first argument is a Site, create a Page on that Site @@ -1230,13 +1230,13 @@ @deprecate_arg("followRedirects", None) @deprecate_arg("loose", None) def imagelinks(self, step=None, total=None, content=False): - """Iterate ImagePage objects for images displayed on this Page. + """Iterate FilePage objects for images displayed on this Page. @param step: limit each API call to this number of pages @param total: iterate no more than this number of pages in total @param content: if True, retrieve the content of the current version of each image description page (default False) - @return: a generator that yields ImagePage objects. + @return: a generator that yields FilePage objects. """ return self.site.pageimages(self, step=step, total=total, @@ -1805,23 +1805,11 @@ pywikibot.warning(u"Page.replaceImage() is no longer supported.") -class ImagePage(Page): +class FilePage(Page): - """A subclass of Page representing an image descriptor wiki page. + """A subclass of Page representing a file description page. - Supports the same interface as Page, with the following added methods: - - getImagePageHtml : Download image page and return raw HTML text. - fileURL : Return the URL for the image described on this - page. - fileIsShared : Return True if image stored on a shared - repository like Wikimedia Commons or Wikitravel. - getFileMd5Sum : Return image file's MD5 checksum. - getFileVersionHistory : Return the image file's version history. - getFileVersionHistoryTable: Return the version history in the form of a - wiki table. - usingPages : Iterate Pages on which the image is displayed. - + Supports the same interface as Page, with some added methods. """ @deprecate_arg("insite", None) @@ -1829,14 +1817,14 @@ """Constructor.""" Page.__init__(self, source, title, 6) if self.namespace() != 6: - raise ValueError(u"'%s' is not in the image namespace!" % title) + raise ValueError(u"'%s' is not in the file namespace!" % title) def getImagePageHtml(self): """ - Download the image page, and return the HTML, as a unicode string. + Download the file page, and return the HTML, as a unicode string. Caches the HTML code, so that if you run this method twice on the - same ImagePage object, the page will only be downloaded once. + same FilePage object, the page will only be downloaded once. """ if not hasattr(self, '_imagePageHtml'): from pywikibot.comms import http @@ -1846,7 +1834,7 @@ return self._imagePageHtml def fileUrl(self): - """Return the URL for the image described on this page.""" + """Return the URL for the file described on this page.""" # TODO add scaling option? if not hasattr(self, '_imageinfo'): self._imageinfo = self.site.loadimageinfo(self) @@ -1861,7 +1849,7 @@ return self.fileIsShared() def fileIsShared(self): - """Check if the image is stored on any known shared repository. + """Check if the file is stored on any known shared repository. @return: bool """ @@ -1878,7 +1866,7 @@ return self.fileUrl().startswith( 'https://upload.wikimedia.org/wikipedia/commons/') - @deprecated("ImagePage.getFileSHA1Sum()") + @deprecated("FilePage.getFileSHA1Sum()") def getFileMd5Sum(self): """Return image file's MD5 checksum.""" # FIXME: MD5 might be performed on incomplete file due to server disconnection @@ -1892,13 +1880,13 @@ return md5Checksum def getFileSHA1Sum(self): - """Return image file's SHA1 checksum.""" + """Return the file's SHA1 checksum.""" if not hasattr(self, '_imageinfo'): self._imageinfo = self.site.loadimageinfo(self) return self._imageinfo['sha1'] def getFileVersionHistory(self): - """Return the image file's version history. + """Return the file's version history. @return: An iterator yielding tuples containing (timestamp, username, resolution, filesize, comment). @@ -1921,7 +1909,7 @@ u'\n|----\n'.join(lines) + '\n|}' def usingPages(self, step=None, total=None, content=False): - """Yield Pages on which the image is displayed. + """Yield Pages on which the file is displayed. @param step: limit each API call to this number of pages @param total: iterate no more than this number of pages in total @@ -1931,6 +1919,9 @@ """ return self.site.imageusage( self, step=step, total=total, content=content) + + +ImagePage = FilePage class Category(Page): @@ -2606,7 +2597,7 @@ raise StopIteration for item in self.site.logevents( logtype='upload', user=self.username, total=total): - yield (ImagePage(self.site, item.title().title()), + yield (FilePage(self.site, item.title().title()), unicode(item.timestamp()), item.comment(), item.pageid() > 0 @@ -3085,7 +3076,7 @@ types = {'wikibase-item': ItemPage, 'string': basestring, - 'commonsMedia': ImagePage, + 'commonsMedia': FilePage, 'globe-coordinate': pywikibot.Coordinate, 'url': basestring, 'time': pywikibot.WbTime, @@ -3254,7 +3245,7 @@ if claim.type == 'wikibase-item': claim.target = ItemPage(site, 'Q' + str(value['numeric-id'])) elif claim.type == 'commonsMedia': - claim.target = ImagePage(site.image_repository(), value) + claim.target = FilePage(site.image_repository(), value) elif claim.type == 'globe-coordinate': claim.target = pywikibot.Coordinate.fromWikibase(value, site) elif claim.type == 'time': diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py index 0220107..3a8ed2c 100644 --- a/pywikibot/pagegenerators.py +++ b/pywikibot/pagegenerators.py @@ -311,12 +311,12 @@ 'pywikibot-enter-file-links-processing') if fileLinksPageTitle.startswith(self.site.namespace(6) + ":"): - fileLinksPage = pywikibot.ImagePage(self.site, - fileLinksPageTitle) + fileLinksPage = pywikibot.FilePage(self.site, + fileLinksPageTitle) else: - fileLinksPage = pywikibot.ImagePage(self.site, - 'Image:' + - fileLinksPageTitle) + fileLinksPage = pywikibot.FilePage(self.site, + 'Image:' + + fileLinksPageTitle) gen = FileLinksGenerator(fileLinksPage) elif arg.startswith('-unusedfiles'): if len(arg) == 12: @@ -663,8 +663,8 @@ yield pywikibot.Page(pywikibot.Link(item["title"], site)) -def FileLinksGenerator(referredImagePage, step=None, total=None, content=False): - return referredImagePage.usingPages(step=step, total=total, content=content) +def FileLinksGenerator(referredFilePage, step=None, total=None, content=False): + return referredFilePage.usingPages(step=step, total=total, content=content) def ImagesPageGenerator(pageWithImages, step=None, total=None, content=False): @@ -956,14 +956,17 @@ yield pywikibot.Category(page) -def ImageGenerator(generator): +def FileGenerator(generator): """ - Wraps around another generator. Yields the same pages, but as ImagePage + Wraps around another generator. Yields the same pages, but as FilePage objects instead of Page objects. Makes sense only if it is ascertained that only images are being retrieved. """ for page in generator: - yield pywikibot.ImagePage(page) + yield pywikibot.FilePage(page) + + +ImageGenerator = FileGenerator def PageWithTalkPageGenerator(generator): @@ -1064,7 +1067,7 @@ if site is None: site = pywikibot.Site() for page in site.unusedfiles(total=total): - yield pywikibot.ImagePage(page.site, page.title()) + yield pywikibot.FilePage(page.site, page.title()) @deprecate_arg("number", "total") diff --git a/pywikibot/site.py b/pywikibot/site.py index 30b07c9..b319a99 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -2743,7 +2743,7 @@ total=None, content=False): """Iterate all images, ordered by image title. - Yields ImagePages, but these pages need not exist on the wiki. + Yields FilePages, but these pages need not exist on the wiki. @param start: start at this title (name need not exist) @param prefix: only iterate titles starting with this substring @@ -2835,11 +2835,11 @@ def imageusage(self, image, namespaces=None, filterredir=None, step=None, total=None, content=False): - """Iterate Pages that contain links to the given ImagePage. + """Iterate Pages that contain links to the given FilePage. - @param image: the image to search for (ImagePage need not exist on + @param image: the image to search for (FilePage need not exist on the wiki) - @type image: ImagePage + @type image: FilePage @param filterredir: if True, only yield redirects; if False (and not None), only yield non-redirects (default: yield both) @param content: if True, load the current content of each iterated page @@ -3812,22 +3812,23 @@ def getImagesFromAnHash(self, hash_found=None): return self.getFilesFromAnHash(hash_found) - def upload(self, imagepage, source_filename=None, source_url=None, + @deprecate_arg('imagepage', 'filepage') + def upload(self, filepage, source_filename=None, source_url=None, comment=None, text=None, watch=False, ignore_warnings=False): """Upload a file to the wiki. Either source_filename or source_url, but not both, must be provided. - @param imagepage: an ImagePage object from which the wiki-name of the + @param filepage: a FilePage object from which the wiki-name of the file will be obtained. @param source_filename: path to the file to be uploaded @param source_url: URL of the file to be uploaded @param comment: Edit summary; if this is not provided, then - imagepage.text will be used. An empty summary is not permitted. + filepage.text will be used. An empty summary is not permitted. This may also serve as the initial page text (see below). @param text: Initial page text; if this is not set, then - imagepage.text will be used, or comment. - @param watch: If true, add imagepage to the bot user's watchlist + filepage.text will be used, or comment. + @param watch: If true, add filepage to the bot user's watchlist @param ignore_warnings: if true, ignore API warnings and force upload (for example, to overwrite an existing file); default False @@ -3855,15 +3856,15 @@ raise ValueError("APISite.upload: must provide either " "source_filename or source_url, not both.") if comment is None: - comment = imagepage.text + comment = filepage.text if not comment: raise ValueError("APISite.upload: cannot upload file without " "a summary/description.") if text is None: - text = imagepage.text + text = filepage.text if not text: text = comment - token = self.token(imagepage, "edit") + token = self.token(filepage, "edit") if source_filename: # upload local file # make sure file actually exists @@ -3873,7 +3874,7 @@ # TODO: if file size exceeds some threshold (to be determined), # upload by chunks (--> os.path.getsize(source_filename)) req = api.Request(site=self, action="upload", token=token, - filename=imagepage.title(withNamespace=False), + filename=filepage.title(withNamespace=False), file=source_filename, comment=comment, text=text, mime=True) else: @@ -3883,7 +3884,7 @@ "User '%s' is not authorized to upload by URL on site %s." % (self.user(), self)) req = api.Request(site=self, action="upload", token=token, - filename=imagepage.title(withNamespace=False), + filename=filepage.title(withNamespace=False), url=source_url, comment=comment, text=text) if watch: req["watch"] = "" @@ -3905,7 +3906,7 @@ pywikibot.output(u"Upload: unrecognized response: %s" % result) if result["result"] == "Success": pywikibot.output(u"Upload successful.") - imagepage._imageinfo = result["imageinfo"] + filepage._imageinfo = result["imageinfo"] return @deprecate_arg("number", "step") @@ -3952,13 +3953,11 @@ yield (newpage, pageitem['timestamp'], pageitem['newlen'], u'', pageitem['user'], pageitem['comment']) - @deprecate_arg("number", None) - @deprecate_arg("repeat", None) - def newimages(self, user=None, start=None, end=None, reverse=False, - step=None, total=None): - """Yield information about newly uploaded images. + def newfiles(self, user=None, start=None, end=None, reverse=False, + step=None, total=None): + """Yield information about newly uploaded files. - Yields a tuple of ImagePage, Timestamp, user(unicode), comment(unicode). + Yields a tuple of FilePage, Timestamp, user(unicode), comment(unicode). N.B. the API does not provide direct access to Special:Newimages, so this is derived from the "upload" log events instead. @@ -3969,11 +3968,17 @@ start=start, end=end, reverse=reverse, step=step, total=total): # event.title() actually returns a Page - image = pywikibot.ImagePage(event.title()) + filepage = pywikibot.FilePage(event.title()) date = event.timestamp() user = event.user() comment = event.comment() or u'' - yield (image, date, user, comment) + yield (filepage, date, user, comment) + + @deprecated("Site().newfiles()") + @deprecate_arg("number", None) + @deprecate_arg("repeat", None) + def newimages(self, *args, **kwargs): + return self.newfiles(*args, **kwargs) @deprecate_arg("number", None) @deprecate_arg("repeat", None) @@ -4069,7 +4074,7 @@ @deprecate_arg("repeat", None) def uncategorizedimages(self, number=None, repeat=True, step=None, total=None): - """Yield ImagePages from Special:Uncategorizedimages.""" + """Yield FilePages from Special:Uncategorizedimages.""" uigen = self._generator(api.ImagePageGenerator, type_arg="querypage", gqppage="Uncategorizedimages", @@ -4111,18 +4116,19 @@ step=step, total=total) return ucgen - @deprecate_arg("number", None) - @deprecate_arg("repeat", None) def unusedfiles(self, step=None, total=None): - """Yield ImagePage objects from Special:Unusedimages.""" + """Yield FilePage objects from Special:Unusedimages.""" uigen = self._generator(api.ImagePageGenerator, type_arg="querypage", gqppage="Unusedimages", step=step, total=total) return uigen - # synonym - unusedimages = unusedfiles + @deprecated("Site().unusedfiles()") + @deprecate_arg("number", None) + @deprecate_arg("repeat", None) + def unusedimages(self, *args, **kwargs): + return self.unusedfiles(*args, **kwargs) @deprecate_arg("number", None) @deprecate_arg("repeat", None) diff --git a/scripts/checkimages.py b/scripts/checkimages.py index 75bff69..c1ba117 100644 --- a/scripts/checkimages.py +++ b/scripts/checkimages.py @@ -637,7 +637,7 @@ """ self.imageName = imageName - self.image = pywikibot.ImagePage(self.site, self.imageName) + self.image = pywikibot.FilePage(self.site, self.imageName) self.timestamp = None self.uploader = None @@ -705,7 +705,7 @@ """ # Get the image's description - reportPageObject = pywikibot.ImagePage(self.site, self.image_to_report) + reportPageObject = pywikibot.FilePage(self.site, self.image_to_report) try: reportPageText = reportPageObject.get() @@ -847,7 +847,7 @@ text) if results: for result in results: - wikiPage = pywikibot.ImagePage(self.site, result) + wikiPage = pywikibot.FilePage(self.site, result) yield wikiPage else: pywikibot.output(link) @@ -863,7 +863,7 @@ regex = re.compile(r'%s' % regexp, re.UNICODE | re.DOTALL) results = regex.findall(textrun) for image in results: - yield pywikibot.ImagePage(self.site, image) + yield pywikibot.FilePage(self.site, image) def loadHiddenTemplates(self): """ Function to load the white templates """ @@ -891,7 +891,7 @@ max_usage = 0 for element in listGiven: imageName = element[1] - imagePage = pywikibot.ImagePage(self.site, imageName) + imagePage = pywikibot.FilePage(self.site, imageName) imageUsage = [page for page in imagePage.usingPages()] if len(imageUsage) > 0 and len(imageUsage) > max_usage: max_usage = len(imageUsage) @@ -1009,7 +1009,7 @@ duplicates_comment_image) duplicateRegex = r'\[\[:File:%s\]\] has the following duplicates' \ % re.escape(self.convert_to_url(self.imageName)) - imagePage = pywikibot.ImagePage(self.site, self.imageName) + imagePage = pywikibot.FilePage(self.site, self.imageName) hash_found = imagePage.getHash() duplicates = self.site.getFilesFromAnHash(hash_found) @@ -1029,7 +1029,7 @@ time_list = [] for duplicate in duplicates: - DupePage = pywikibot.ImagePage(self.site, duplicate) + DupePage = pywikibot.FilePage(self.site, duplicate) if DupePage.title(asUrl=True) != self.image.title(asUrl=True) or \ self.timestamp is None: @@ -1040,16 +1040,16 @@ time_list.append(data_seconds) older_image = self.returnOlderTime(time_image_list, time_list) # And if the images are more than two? - Page_oder_image = pywikibot.ImagePage(self.site, older_image) + Page_oder_image = pywikibot.FilePage(self.site, older_image) string = '' images_to_tag_list = [] for duplicate in duplicates: - if pywikibot.ImagePage(self.site, duplicate) \ - == pywikibot.ImagePage(self.site, older_image): + if pywikibot.FilePage(self.site, duplicate) \ + == pywikibot.FilePage(self.site, older_image): # the older image, not report also this as duplicate continue - DupePage = pywikibot.ImagePage(self.site, duplicate) + DupePage = pywikibot.FilePage(self.site, duplicate) try: DupPageText = DupePage.get() older_page_text = Page_oder_image.get() diff --git a/scripts/harvest_template.py b/scripts/harvest_template.py index ac9bc01..1f09720 100755 --- a/scripts/harvest_template.py +++ b/scripts/harvest_template.py @@ -172,9 +172,9 @@ elif claim.type == 'commonsMedia': commonssite = pywikibot.Site("commons", "commons") imagelink = pywikibot.Link(value, source=commonssite, defaultNamespace=6) - image = pywikibot.ImagePage(imagelink) + image = pywikibot.FilePage(imagelink) if image.isRedirectPage(): - image = pywikibot.ImagePage(image.getRedirectTarget()) + image = pywikibot.FilePage(image.getRedirectTarget()) if not image.exists(): pywikibot.output('[[%s]] doesn\'t exist so I can\'t link to it' % (image.title(),)) continue diff --git a/scripts/illustrate_wikidata.py b/scripts/illustrate_wikidata.py index 01504de..cf6cdaf 100644 --- a/scripts/illustrate_wikidata.py +++ b/scripts/illustrate_wikidata.py @@ -67,9 +67,9 @@ newclaim = pywikibot.Claim(self.repo, self.wdproperty) commonssite = pywikibot.Site("commons", "commons") imagelink = pywikibot.Link(imagename, source=commonssite, defaultNamespace=6) - image = pywikibot.ImagePage(imagelink) + image = pywikibot.FilePage(imagelink) if image.isRedirectPage(): - image = pywikibot.ImagePage(image.getRedirectTarget()) + image = pywikibot.FilePage(image.getRedirectTarget()) if not image.exists(): pywikibot.output('[[%s]] doesn\'t exist so I can\'t link to it' % (image.title(),)) continue diff --git a/scripts/image.py b/scripts/image.py index 3189d71..9d2ffc8 100644 --- a/scripts/image.py +++ b/scripts/image.py @@ -191,7 +191,7 @@ if old_image: site = pywikibot.Site() - old_imagepage = pywikibot.ImagePage(site, old_image) + old_imagepage = pywikibot.FilePage(site, old_image) gen = pagegenerators.FileLinksGenerator(old_imagepage) preloadingGen = pagegenerators.PreloadingGenerator(gen) bot = ImageRobot(preloadingGen, old_image, new_image, **options) diff --git a/scripts/imagerecat.py b/scripts/imagerecat.py index 2956bc1..88c2318 100644 --- a/scripts/imagerecat.py +++ b/scripts/imagerecat.py @@ -76,7 +76,7 @@ for page in generator: if page.exists() and (page.namespace() == 6) and \ (not page.isRedirectPage()): - imagepage = pywikibot.ImagePage(page.site, page.title()) + imagepage = pywikibot.FilePage(page.site, page.title()) pywikibot.output(u'Working on ' + imagepage.title()) if onlyUncat and not(u'Uncategorized' in imagepage.templates()): diff --git a/scripts/imagetransfer.py b/scripts/imagetransfer.py index aff1a39..2706554 100644 --- a/scripts/imagetransfer.py +++ b/scripts/imagetransfer.py @@ -265,7 +265,7 @@ for linkedPage in page.interwiki(): imagelist.append(linkedPage.imagelinks(followRedirects=True)) elif page.isImage(): - imagePage = pywikibot.ImagePage(page.site, page.title()) + imagePage = pywikibot.FilePage(page.site, page.title()) imagelist = [imagePage] else: imagePage = (page.imagelinks(followRedirects=True)).result( diff --git a/scripts/maintenance/compat2core.py b/scripts/maintenance/compat2core.py index cad299f..bc74cf3 100644 --- a/scripts/maintenance/compat2core.py +++ b/scripts/maintenance/compat2core.py @@ -4,9 +4,9 @@ This is a helper script to convert compat 1.0 scripts to the new core 2.0 framework. -NOTE: Please be aware that this script is not be able to convert your codes +NOTE: Please be aware that this script is not able to convert your codes completely. It may support you with some automatic replacements and it gives -some warnings and hints for converting. Please refer the converting guide +some warnings and hints for converting. Please refer to the converting guide README-conversion.txt in the core framework folder and check your codes finally. The scripts asks for the .py file and converts it to @@ -36,7 +36,7 @@ import codecs import pywikibot -# be carefull with replacement order! +# be careful with replacement order! replacements = ( # doc strings ('#\r?\n__version__', @@ -63,6 +63,8 @@ ('catlib\.change_category\s*$(\s*)(?P<article>.+?),\s*(?P<oldcat>.+?),', r'\g<article>.change_category(\1\g<oldcat>,'), ('userlib\.User\s*\(\s*', 'pywikibot.User('), + # change ImagePage to FilePage + ('pywikibot\.ImagePage\s*\(\s*', 'pywikibot.FilePage('), # deprecated title methods ('\.urlname\s*\(\s*$', '.title(asUrl=True)'), ('\.urlname\s*$\s*(?:withNamespace\s*=\s*)?(True|False)+\s*$', @@ -72,9 +74,10 @@ ('\.aslink\s*$\s*$', '.title(asLink=True)'), # other deprecated methods ('(?<!site)\.encoding\s*$\s*$', '.site.encoding()'), + ('\.newimages\s*$', '.newfiles('), # new core methods ('\.get\s*\(\s*get_redirect\s*=\s*True\s*$', '.text'), - # stopme() is doen by the framework itself + # stopme() is done by the framework itself ('(\s*)try\:\s*\r?\n\s+main\s*\r?\n\s*finally\:\s*\r?\n\s+pywikibot\.stopme', r'\1main()'), ) @@ -95,7 +98,7 @@ 'User.contributions() returns a pywikibot.Timestamp object instead of a\n' 'MediaWiki one'), ('.getFileMd5Sum(', - 'ImagePage.getFileMd5Sum() is deprecated should be replaced by ' + 'FilePage.getFileMd5Sum() is deprecated should be replaced by ' 'getFileSHA1Sum()'), (' wikipedia.', '"wikipedia" library has been changed to "pywikibot".'), diff --git a/scripts/nowcommons.py b/scripts/nowcommons.py index 19de9ac..14c3f06 100644 --- a/scripts/nowcommons.py +++ b/scripts/nowcommons.py @@ -320,7 +320,7 @@ pywikibot.output(u"\n\n>>> \03{lightpurple}%s\03{default} <<<" % page.title()) try: - localImagePage = pywikibot.ImagePage(self.site, page.title()) + localImagePage = pywikibot.FilePage(self.site, page.title()) if localImagePage.fileIsShared(): pywikibot.output(u'File is already on Commons.') continue @@ -333,7 +333,7 @@ if not filenameOnCommons and not self.getOption('use_hash'): pywikibot.output(u'NowCommons template not found.') continue - commonsImagePage = pywikibot.ImagePage(commons, 'Image:%s' + commonsImagePage = pywikibot.FilePage(commons, 'Image:%s' % filenameOnCommons) if localImagePage.title(withNamespace=False) == \ commonsImagePage.title(withNamespace=False) and self.getOption('use_hash'): @@ -362,7 +362,7 @@ oImageRobot.run() # If the image is used with the urlname the # previous function won't work - if len(list(pywikibot.ImagePage(self.site, + if len(list(pywikibot.FilePage(self.site, page.title()).usingPages())) > 0 and \ self.getOption('replaceloose'): oImageRobot = image.ImageRobot( @@ -376,7 +376,7 @@ self.getOption('replaceloose')) oImageRobot.run() # refresh because we want the updated list - usingPages = len(list(pywikibot.ImagePage( + usingPages = len(list(pywikibot.FilePage( self.site, page.title()).usingPages())) if usingPages > 0 and self.getOption('use_hash'): # just an enter diff --git a/scripts/upload.py b/scripts/upload.py index 8a694ae..5c39065 100755 --- a/scripts/upload.py +++ b/scripts/upload.py @@ -198,7 +198,7 @@ filename = self.process_filename() site = self.targetSite - imagepage = pywikibot.ImagePage(site, filename) # normalizes filename + imagepage = pywikibot.FilePage(site, filename) # normalizes filename imagepage.text = self.description pywikibot.output(u'Uploading file to %s via API....' % site) diff --git a/tests/page_tests.py b/tests/page_tests.py index 4948638..dca5d2f 100644 --- a/tests/page_tests.py +++ b/tests/page_tests.py @@ -352,7 +352,7 @@ for p in mainpage.langlinks(): self.assertType(p, pywikibot.Link) for p in mainpage.imagelinks(): - self.assertType(p, pywikibot.ImagePage) + self.assertType(p, pywikibot.FilePage) for p in mainpage.templates(): self.assertType(p, pywikibot.Page) for t, params in mainpage.templatesWithParams(): diff --git a/tests/site_tests.py b/tests/site_tests.py index 621da4e..241bcbe 100644 --- a/tests/site_tests.py +++ b/tests/site_tests.py @@ -312,7 +312,7 @@ for cm in mysite.categorymembers(cat): self.assertType(cat, pywikibot.Page) # test pageimages - self.assertTrue(all(isinstance(im, pywikibot.ImagePage) + self.assertTrue(all(isinstance(im, pywikibot.FilePage) for im in mysite.pageimages(mainpage))) # test pagetemplates self.assertTrue(all(isinstance(te, pywikibot.Page) @@ -473,27 +473,27 @@ ai = list(mysite.allimages(total=10)) self.assertTrue(len(ai) <= 10) - self.assertTrue(all(isinstance(image, pywikibot.ImagePage) + self.assertTrue(all(isinstance(image, pywikibot.FilePage) for image in ai)) for impage in mysite.allimages(start="Ba", total=5): - self.assertType(impage, pywikibot.ImagePage) + self.assertType(impage, pywikibot.FilePage) self.assertTrue(mysite.page_exists(impage)) self.assertTrue(impage.title(withNamespace=False) >= "Ba") # # Bug # 15985 # for impage in mysite.allimages(start="Da", reverse=True, total=5): -# self.assertType(impage, pywikibot.ImagePage) +# self.assertType(impage, pywikibot.FilePage) # self.assertTrue(mysite.page_exists(impage)) # self.assertTrue(impage.title() <= "Da") for impage in mysite.allimages(prefix="Ch", total=5): - self.assertType(impage, pywikibot.ImagePage) + self.assertType(impage, pywikibot.FilePage) self.assertTrue(mysite.page_exists(impage)) self.assertTrue(impage.title(withNamespace=False).startswith("Ch")) for impage in mysite.allimages(minsize=100, total=5): - self.assertType(impage, pywikibot.ImagePage) + self.assertType(impage, pywikibot.FilePage) self.assertTrue(mysite.page_exists(impage)) self.assertTrue(impage._imageinfo["size"] >= 100) for impage in mysite.allimages(maxsize=2000, total=5): - self.assertType(impage, pywikibot.ImagePage) + self.assertType(impage, pywikibot.FilePage) self.assertTrue(mysite.page_exists(impage)) self.assertTrue(impage._imageinfo["size"] <= 2000) -- To view, visit https://gerrit.wikimedia.org/r/152894 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I289da42fa4b5f78e4837c8e4c1a3a72e9d322990 Gerrit-PatchSet: 8 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Ricordisamoa <ricordisamoa(a)openmailbox.org> Gerrit-Reviewer: Beta16 <l.rabinelli(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: Pyfisch <pyfisch(a)gmail.com> Gerrit-Reviewer: Ricordisamoa <ricordisamoa(a)openmailbox.org> Gerrit-Reviewer: Russell Blau <russblau(a)imapmail.org> Gerrit-Reviewer: Underlying lk <a375070(a)drdrb.net> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] APISite.siteinfo automatically fetches the results - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: APISite.siteinfo automatically fetches the results ...................................................................... APISite.siteinfo automatically fetches the results This changes it from a dict to a Wrapper which automatically queries the result if they aren't cached. It is basically merging APISite._add_siteinfo, APISite._getsiteinfo and APISite.siteinfo. It'll always caches the result of the siprop "general", because otherwise it would be unable to determine if a requested property is in fact a property of "general" (e.g. APISite.case()). It also supports the "in" operator which tells you if the value is cached. Setting it from the outside is not possible anymore. If a property isn't supported (e.g. 'restrictions' before 1.23) it will cache either an empty dummy object (which acts like an empty string/iterable/dict) or a specific default constant. Change-Id: Ib5b6d84f0baf60376cdc25bc715d346c1579da1f --- M pywikibot/data/api.py M pywikibot/site.py M pywikibot/tools.py M tests/dry_api_tests.py M tests/dry_site_tests.py M tests/site_tests.py M tests/utils.py 7 files changed, 378 insertions(+), 147 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py index efe88b3..9ff7ba7 100644 --- a/pywikibot/data/api.py +++ b/pywikibot/data/api.py @@ -157,7 +157,7 @@ # otherwise be a problem. # This situation is only tripped when one of the first actions # on the site is a write action and the extension isn't installed. - if hasattr(self.site, "_extensions"): + if 'extensions' in self.site.siteinfo: use_assert_edit_extension = self.site.hasExtension('AssertEdit', False) else: use_assert_edit_extension = True diff --git a/pywikibot/site.py b/pywikibot/site.py index 63fca60..30b07c9 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -20,13 +20,15 @@ import re import sys from distutils.version import LooseVersion as LV -from collections import Iterable +from collections import Iterable, Container import threading import time import urllib import json +import copy import pywikibot +import pywikibot.tools as tools from pywikibot import deprecate_arg from pywikibot import config from pywikibot import deprecated @@ -890,6 +892,226 @@ return decorator +class Siteinfo(Container): + + """ + A 'dictionary' like container for siteinfo. + + This class queries the server to get the requested siteinfo property. + Optionally it can cache this directly in the instance so that later + requests don't need to query the server. + + All values of the siteinfo property 'general' are directly available. + """ + + def __init__(self, site): + """Initialise it with an empty cache.""" + self._site = site + self._cache = {} + + @staticmethod + def _get_default(key): + """ + Return the default value for different properties. + + If the property is 'restrictions' it returns a dictionary with: + 'cascadinglevels': 'sysop' + 'semiprotectedlevels': 'autoconfirmed' + 'levels': '' (everybody), 'autoconfirmed', 'sysop' + 'types': 'create', 'edit', 'move', 'upload' + Otherwise it returns L{tools.EMPTY_DEFAULT}. + + @param key: The property name + @type key: str + @return: The default value + @rtype: dict or L{tools.EmptyDefault} + """ + if key == 'restrictions': + # implemented in b73b5883d486db0e9278ef16733551f28d9e096d + return { + 'cascadinglevels': ['sysop'], + 'semiprotectedlevels': ['autoconfirmed'], + 'levels': ['', 'autoconfirmed', 'sysop'], + 'types': ['create', 'edit', 'move', 'upload'] + } + else: + return tools.EMPTY_DEFAULT + + def _get_siteinfo(self, prop, force=False): + """ + Retrieve a siteinfo property. All properties which the site doesn't + support contain the default value. Because pre-1.12 no data was + returned when a property doesn't exists, it queries each property + independetly if a property is invalid. + + @param prop: The property names of the siteinfo. + @type prop: str or iterable + @param force: Don't access the cached request. + @type force: bool + @return: A dictionary with the properties of the site. Each entry in + the dictionary is a tuple of the value and a boolean to save if it + is the default value. + @rtype: dict (the values) + @see: U{https://www.mediawiki.org/wiki/API:Meta#siteinfo_.2F_si} + """ + if isinstance(prop, basestring): + props = [prop] + else: + props = prop + if len(props) == 0: + raise ValueError('At least one property name must be provided.') + try: + data = pywikibot.data.api.CachedRequest( + expiry=0 if force else pywikibot.config.API_config_expiry, + site=self._site, + action='query', + meta='siteinfo', + siprop='|'.join(props)).submit() + except api.APIError as e: + if e.code == 'siunknown_siprop': + if len(props) == 1: + pywikibot.log(u"Unable to get siprop '{0}'".format(props[0])) + return {props[0]: (Siteinfo._get_default(props[0]), True)} + else: + pywikibot.log(u"Unable to get siteinfo, because at least " + u"one property is unknown: '{0}'".format( + u"', '".join(props))) + results = {} + for prop in props: + results.update(self._get_siteinfo(prop, force)) + return results + else: + raise + else: + result = {} + if 'warnings' in data: + invalid_properties = [] + for prop in re.match(u"^Unrecognized values? for parameter " + u"'siprop': ([^,]+(?:, [^,]+)*)$", + data['warnings']['siteinfo']['*']).group(1).split(','): + prop = prop.strip() + invalid_properties += [prop] + result[prop] = (Siteinfo._get_default(prop), True) + pywikibot.log(u"Unable to get siprop(s) '{0}'".format( + u"', '".join(invalid_properties))) + if 'query' in data: + # todo iterate through the properties! + for prop in props: + if prop in data['query']: + result[prop] = (data['query'][prop], False) + return result + + def _get_general(self, key, force): + """ + Return a siteinfo property which is loaded by default. + + The property 'general' will be queried if it wasn't yet or it's forced. + Additionally all uncached default properties are queried. This way + multiple default properties are queried with one request. It'll cache + always all results. + + @param key: The key to search for. + @type key: str + @param force: If 'general' should be queried in any case. + @type force: bool + @return: If that property was retrived via this method. Returns an + empty tuple if it wasn't retrieved. + @rtype: various (the value), bool (if the default value is used) + """ + if 'general' not in self._cache: + pywikibot.debug('general siteinfo not loaded yet.', _logger) + force = True + props = ['namespaces', 'namespacealiases'] + else: + props = [] + if force: + props = [prop for prop in props if prop not in self._cache] + if props: + pywikibot.debug( + u"Load siteinfo properties '{0}' along with 'general'".format( + u"', '".join(props)), _logger) + props += ['general'] + default_info = self._get_siteinfo(props, force) + for prop in props: + self._cache[prop] = default_info[prop] + if key in default_info: + return default_info[key] + if key in self._cache['general'][0]: + return self._cache['general'][0][key], False + else: + return tuple() + + def __getitem__(self, key): + """Return a siteinfo property, caching and not forcing it.""" + return self.get(key, False) # caches and doesn't force it + + def get(self, key, get_default=True, cache=True, force=False): + """ + Return a siteinfo property. + + @param key: The name of the siteinfo property. + @type key: str + @param get_default: Whether to throw an KeyError if the key is invalid. + @type get_default: bool + @param cache: Caches the result interally so that future accesses via + this method won't query the server. + @type cache: bool + @param force: Ignores the cache and always queries the server to get + the newest value. + @type force: bool + @return: The gathered property + @rtype: various + @see: L{_get_siteinfo} + """ + if not force: + try: + cached = self._get_cached(key) + except KeyError: + cached = None + # a not recognised result was cached, but they aren't requested + if cached: + if cached[1] and not get_default: + raise KeyError(key) + else: + return copy.deepcopy(cached[0]) + preloaded = self._get_general(key, force) + if not preloaded: + preloaded = self._get_siteinfo(key, force)[key] + else: + cache = False + if preloaded[1] and not get_default: + raise KeyError(key) + else: + if cache: + self._cache[key] = preloaded + return copy.deepcopy(preloaded[0]) + + def _get_cached(self, key): + """Return the cached value or a KeyError exception if not cached.""" + if 'general' in self._cache: + if key in self._cache['general'][0]: + return (self._cache['general'][0][key], False) + else: + return self._cache[key] + raise KeyError(key) + + def __contains__(self, key): + """Return whether the value is cached.""" + try: + self._get_cached(key) + except KeyError: + return False + else: + return True + + def is_recognised(self, key): + """Return if 'key' is a valid property name. 'None' if not cached.""" + try: + return not self._get_cached(key)[1] + except KeyError: + return None + + class APISite(BaseSite): """API interface to MediaWiki site. @@ -924,6 +1146,7 @@ BaseSite.__init__(self, code, fam, user, sysop) self._msgcache = {} self._loginstatus = LoginStatus.NOT_ATTEMPTED + self._siteinfo = Siteinfo(self) return @staticmethod @@ -1067,8 +1290,6 @@ if sysop else LoginStatus.AS_USER) else: self._loginstatus = LoginStatus.NOT_LOGGED_IN # failure - if not hasattr(self, "_siteinfo"): - self._getsiteinfo() forceLogin = login # alias for backward-compatibility @@ -1383,15 +1604,9 @@ def getmagicwords(self, word): """Return list of localized "word" magic words for the site.""" if not hasattr(self, "_magicwords"): - try: - # don't cache in _siteinfo, because we cache it in _magicwords - magicwords = self._add_siteinfo("magicwords", False) - self._magicwords = dict((item["name"], item["aliases"]) + magicwords = self.siteinfo.get("magicwords", cache=False) + self._magicwords = dict((item["name"], item["aliases"]) for item in magicwords) - except api.APIError: - # hack for older sites that don't support 1.13 properties - # probably should delete if we're not going to support pre-1.13 - self._magicwords = {} if word in self._magicwords: return self._magicwords[word] @@ -1432,76 +1647,7 @@ """Return list of localized PAGENAMEE tags for the site.""" return self.getmagicwords("pagenamee") - def _add_siteinfo(self, prop, cache, force=False): - """ - Retrieve additional siteinfo and optionally cache it. - - Queries the site and returns the properties. It can cache the value - so that future queries will access the cache. With C{force} set to - True it won't access the cache but it can still cache the value. If - the property doesn't exists it returns None. - - @param prop: The property name of the siteinfo. - @type prop: str - @param cache: Should this be cached? - @type cache: bool - @param force: Should the cache be skipped? - @type force: bool - @return: The properties of the site. - @rtype: various (depends on prop) - """ - if not hasattr(self, '_siteinfo'): - force = True # if it doesn't exists there won't be a cache - if cache: # but only initialise cache if that is requested - self._getsiteinfo() - if not force and prop in self._siteinfo: - return self._siteinfo[prop] - data = pywikibot.data.api.CachedRequest( - expiry=0 if force else pywikibot.config.API_config_expiry, - site=self, - action='query', - meta='siteinfo', - siprop=prop).submit() - try: - prop_data = data['query'][prop] - except KeyError: - prop_data = None - if cache: - self._siteinfo[prop] = prop_data - return prop_data - - def _getsiteinfo(self, force=False): - """Retrieve siteinfo and namespaces from site.""" - sirequest = api.CachedRequest( - expiry=(0 if force else config.API_config_expiry), - site=self, - action="query", - meta="siteinfo", - siprop="general|namespaces|namespacealiases|extensions" - ) - try: - sidata = sirequest.submit() - except api.APIError: - # hack for older sites that don't support 1.12 properties - # probably should delete if we're not going to support pre-1.12 - sirequest = api.Request( - site=self, - action="query", - meta="siteinfo", - siprop="general|namespaces" - ) - sidata = sirequest.submit() - - assert 'query' in sidata, \ - "API siteinfo response lacks 'query' key" - sidata = sidata['query'] - assert 'general' in sidata, \ - "API siteinfo response lacks 'general' key" - assert 'namespaces' in sidata, \ - "API siteinfo response lacks 'namespaces' key" - self._siteinfo = sidata['general'] - - nsdata = sidata['namespaces'] + def _build_namespaces(self): self._namespaces = {} @@ -1511,36 +1657,27 @@ # the defaults defined in Namespace. is_mw114 = LV(self.version()) >= LV('1.14') - for nskey in nsdata: - ns = int(nskey) + for nsdata in self.siteinfo.get('namespaces', cache=False).values(): + ns = nsdata.pop('id') custom_name = None canonical_name = None if ns == 0: - canonical_name = nsdata[nskey].pop('*') + canonical_name = nsdata.pop('*') custom_name = canonical_name else: - custom_name = nsdata[nskey].pop('*') + custom_name = nsdata.pop('*') if is_mw114: - canonical_name = nsdata[nskey].pop('canonical') + canonical_name = nsdata.pop('canonical') - # Remove the 'id' from nsdata - nsdata[nskey].pop('id') namespace = Namespace(ns, canonical_name, custom_name, - use_image_name=not is_mw114, **nsdata[nskey]) - + use_image_name=not is_mw114, + **nsdata) self._namespaces[ns] = namespace - if 'namespacealiases' in sidata: - aliasdata = sidata['namespacealiases'] - for item in aliasdata: - ns = int(item['id']) - if item['*'] not in self._namespaces[ns]: - self._namespaces[ns].aliases.append(item['*']) - - if 'extensions' in sidata: - self._extensions = sidata['extensions'] - else: - self._extensions = None + for item in self.siteinfo.get('namespacealiases'): + ns = int(item['id']) + if item['*'] not in self._namespaces[ns]: + self._namespaces[ns].aliases.append(item['*']) def hasExtension(self, name, unknown=NotImplementedError): """ Determine whether extension `name` is loaded. @@ -1552,15 +1689,15 @@ @return: bool """ - if not hasattr(self, '_extensions'): - self._getsiteinfo() - if self._extensions is None: + try: + extensions = self.siteinfo['extensions'] + except KeyError: if isinstance(unknown, type) and issubclass(unknown, Exception): raise unknown( "Feature 'hasExtension' only available in MW 1.14+") else: return unknown - for ext in self._extensions: + for ext in extensions: if ext['name'].lower() == name.lower(): return True return False @@ -1568,8 +1705,6 @@ @property def siteinfo(self): """Site information dict.""" - if not hasattr(self, "_siteinfo"): - self._getsiteinfo() return self._siteinfo def case(self): @@ -1632,8 +1767,8 @@ def namespaces(self): """Return dict of valid namespaces on this wiki.""" - if not hasattr(self, "_siteinfo"): - self._getsiteinfo() + if not hasattr(self, '_namespaces'): + self._build_namespaces() return self._namespaces def namespace(self, num, all=False): @@ -1655,9 +1790,7 @@ """ try: - if force: - self._getsiteinfo(force=True) # drop/expire cache and reload - versionstring = self.siteinfo['generator'] + versionstring = self.siteinfo.get('generator', force=force) m = re.match(r"^MediaWiki ([0-9]+)\.([0-9]+)(.*)$", versionstring) if m: return (int(m.group(1)), int(m.group(2)), m.group(3)) @@ -3504,37 +3637,22 @@ """ Return the protection types available on this site. - With MediaWiki version 1.23 protection types can be retrieved. To - support older wikis, the default protection types 'create', 'edit', - 'move' and 'upload' are returned. - @return protection types available @rtype: set of unicode instances + @see: L{Siteinfo._get_default()} """ - # implemented in b73b5883d486db0e9278ef16733551f28d9e096d - restrictions = self._add_siteinfo('restrictions', True) - if restrictions is None or 'types' not in restrictions: - return set([u'create', u'edit', u'move', u'upload']) - else: - return set(restrictions['types']) + return set(self.siteinfo.get('restrictions')['types']) def protection_levels(self): """ Return the protection levels available on this site. - With MediaWiki version 1.23 protection levels can be retrieved. To - support older wikis, the default protection levels '', 'autoconfirmed', - and 'sysop' are returned. - @return protection types available @rtype: set of unicode instances + @see: L{Siteinfo._get_default()} """ # implemented in b73b5883d486db0e9278ef16733551f28d9e096d - restrictions = self._add_siteinfo('restrictions', True) - if restrictions is None or 'levels' not in restrictions: - return set([u'', u'autoconfirmed', u'sysop']) - else: - return set(restrictions['levels']) + return set(self.siteinfo.get('restrictions')['levels']) @must_be(group='sysop') @deprecate_arg("summary", "reason") diff --git a/pywikibot/tools.py b/pywikibot/tools.py index 6f0c26c..bf3dbc9 100644 --- a/pywikibot/tools.py +++ b/pywikibot/tools.py @@ -10,6 +10,7 @@ import sys import threading import time +from collections import Mapping if sys.version_info[0] > 2: import queue as Queue @@ -216,6 +217,44 @@ thd.start() +class CombinedError(KeyError, IndexError): + + """An error that gets caught by both KeyError and IndexError.""" + + +class EmptyDefault(str, Mapping): + + """ + A default for a not existing siteinfo property. + + It should be chosen if there is no better default known. It acts like an + empty collections, so it can be iterated through it savely if treated as a + list, tuple, set or dictionary. It is also basically an empty string. + + Accessing a value via __getitem__ will result in an combined KeyError and + IndexError. + """ + + def __init__(self): + """Initialise the default as an empty string.""" + str.__init__(self) + + # http://stackoverflow.com/a/13243870/473890 + def _empty_iter(self): + """An iterator which does nothing.""" + return + yield + + def __getitem__(self, key): + """Raise always a L{CombinedError}.""" + raise CombinedError(key) + + iteritems = itervalues = iterkeys = __iter__ = _empty_iter + + +EMPTY_DEFAULT = EmptyDefault() + + if __name__ == "__main__": def _test(): import doctest diff --git a/tests/dry_api_tests.py b/tests/dry_api_tests.py index c6cddf8..6c61a33 100644 --- a/tests/dry_api_tests.py +++ b/tests/dry_api_tests.py @@ -10,7 +10,7 @@ import datetime import pywikibot from pywikibot.data.api import CachedRequest, QueryGenerator -from utils import unittest, NoSiteTestCase, SiteTestCase +from utils import unittest, NoSiteTestCase, SiteTestCase, DummySiteinfo class DryCachedRequestTests(SiteTestCase): @@ -68,6 +68,7 @@ def __init__(self): self._user = 'anon' pywikibot.site.BaseSite.__init__(self, 'mock', MockFamily()) + self._siteinfo = DummySiteinfo({'case': 'first-letter'}) def version(self): return '1.13' # pre 1.14 @@ -84,9 +85,9 @@ def encodings(self): return [] - def _getsiteinfo(self): - self._siteinfo = {'case': 'first-letter'} - return {} + @property + def siteinfo(self): + return self._siteinfo def __repr__(self): return "MockSite()" diff --git a/tests/dry_site_tests.py b/tests/dry_site_tests.py index 709d569..ebea4ce 100644 --- a/tests/dry_site_tests.py +++ b/tests/dry_site_tests.py @@ -10,7 +10,7 @@ import pywikibot from pywikibot.site import must_be -from tests.utils import unittest, NoSiteTestCase +from tests.utils import unittest, NoSiteTestCase, DummySiteinfo class DrySite(pywikibot.site.APISite): @@ -20,6 +20,10 @@ def userinfo(self): return self._userinfo + @property + def siteinfo(self): + return DummySiteinfo({}) + class TestDrySite(NoSiteTestCase): def test_logged_in(self): diff --git a/tests/site_tests.py b/tests/site_tests.py index 171bc4e..621da4e 100644 --- a/tests/site_tests.py +++ b/tests/site_tests.py @@ -14,6 +14,8 @@ from collections import Iterable import pywikibot from tests.utils import PywikibotTestCase, unittest +from datetime import datetime +import re import sys if sys.version_info[0] > 2: @@ -196,12 +198,7 @@ self.assertFalse('*' in mysite.mediawiki_messages(['*'])) self.assertType(mysite.getcurrenttimestamp(), basestring) - self.assertType(mysite.siteinfo, dict) - self.assertType(mysite.case(), basestring) - ver = mysite.live_version() - self.assertType(ver, tuple) - self.assertTrue(all(isinstance(ver[i], int) for i in (0, 1))) - self.assertType(ver[2], basestring) + self.assertType(mysite.siteinfo, pywikibot.site.Siteinfo) self.assertType(mysite.months_names, list) self.assertEqual(mysite.months_names[4], (u'May', u'May')) self.assertEqual(mysite.list_to_text(('pywikibot',)), 'pywikibot') @@ -980,8 +977,9 @@ # and the other following methods in site.py def testExtensions(self): - # test automatically getting _extensions - del mysite._extensions + # test automatically getting extensions cache + if 'extensions' in mysite.siteinfo: + del mysite.siteinfo._cache['extensions'] self.assertTrue(mysite.hasExtension('Disambiguator')) # test case-sensitivity @@ -990,7 +988,7 @@ self.assertFalse(mysite.hasExtension('ThisExtensionDoesNotExist')) # test behavior for sites that do not report extensions - mysite._extensions = None + mysite.siteinfo._cache['extensions'] = (None, True) self.assertRaises(NotImplementedError, mysite.hasExtension, ('anything')) class MyException(Exception): @@ -999,7 +997,7 @@ self.assertTrue(mysite.hasExtension('anything', True)) self.assertFalse(mysite.hasExtension('anything', False)) - del mysite._extensions + del mysite.siteinfo._cache['extensions'] def test_API_limits_with_site_methods(self): # test step/total parameters for different sitemethods @@ -1027,6 +1025,47 @@ pickle.dumps(site) self.assertTrue(True) # No exception thrown! + def testSiteinfo(self): + """Test the siteinfo property.""" + # general enteries + self.assertIsInstance(mysite.siteinfo['timeoffset'], (int, float)) + self.assertTrue(-12 * 60 <= mysite.siteinfo['timeoffset'] <= +14 * 60) + self.assertEqual(mysite.siteinfo['timeoffset'] % 15, 0) + self.assertRegexpMatches(mysite.siteinfo['timezone'], "([A-Z]{3,4}|[A-Z][a-z]+/[A-Z][a-z]+)") + self.assertType(datetime.strptime(mysite.siteinfo['time'], "%Y-%m-%dT%H:%M:%SZ"), datetime) + self.assertTrue(mysite.siteinfo['maxuploadsize'] > 0) + self.assertIn(mysite.case(), ["first-letter", "case-sensitive"]) + self.assertEqual(re.findall("\$1", mysite.siteinfo['articlepath']), ["$1"]) + ver = mysite.live_version() + self.assertType(ver, tuple) + self.assertTrue(all(isinstance(ver[i], int) for i in (0, 1))) + self.assertType(ver[2], basestring) + + def entered_loop(iterable): + for iterable_item in iterable: + return True + return False + + self.assertType(mysite.siteinfo.get('restrictions'), dict) + self.assertTrue('restrictions' in mysite.siteinfo) + # the following line only works in 1.23+ + self.assertTrue(mysite.siteinfo.is_recognised('restrictions')) + del mysite.siteinfo._cache['restrictions'] + self.assertType(mysite.siteinfo.get('restrictions', cache=False), dict) + self.assertFalse('restrictions' in mysite.siteinfo) + + not_exists = 'this-property-does-not-exist' + self.assertRaises(KeyError, mysite.siteinfo.__getitem__, not_exists) + self.assertFalse(not_exists in mysite.siteinfo) + self.assertEqual(len(mysite.siteinfo.get(not_exists)), 0) + self.assertFalse(entered_loop(mysite.siteinfo.get(not_exists))) + self.assertFalse(entered_loop(mysite.siteinfo.get(not_exists).iteritems())) + self.assertFalse(entered_loop(mysite.siteinfo.get(not_exists).itervalues())) + self.assertFalse(entered_loop(mysite.siteinfo.get(not_exists).iterkeys())) + self.assertFalse(entered_loop(mysite.siteinfo.get(not_exists).items())) + self.assertFalse(entered_loop(mysite.siteinfo.get(not_exists).values())) + self.assertFalse(entered_loop(mysite.siteinfo.get(not_exists).keys())) + class TestSiteLoadRevisions(PywikibotTestCase): """Test cases for Site.loadrevision() method.""" diff --git a/tests/utils.py b/tests/utils.py index f767538..db30699 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -113,3 +113,33 @@ unpatch_request() PywikibotTestCase = CachedTestCase + + +class DummySiteinfo(): + + def __init__(self, cache): + self._cache = dict((key, (item, False)) for key, item in cache.iteritems()) + + def __getitem__(self, key): + return self.get(key, False) + + def get(self, key, get_default=True, cache=True, force=False): + if not force and key in self._cache: + loaded = self._cache[key] + if not loaded[1] and not get_default: + raise KeyError(key) + else: + return loaded[0] + elif get_default: + default = pywikibot.site.Siteinfo._get_default(key) + if cache: + self._cache[key] = (default, True) + return default + else: + raise KeyError(key) + + def __contains__(self, key): + return False + + def is_recognised(self, key): + return None -- To view, visit https://gerrit.wikimedia.org/r/154413 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ib5b6d84f0baf60376cdc25bc715d346c1579da1f Gerrit-PatchSet: 12 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] Re-enable voting 2.6 tests - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Re-enable voting 2.6 tests ...................................................................... Re-enable voting 2.6 tests Let's keep them green, people! Change-Id: If3f6414f48a10b995b43b9e2826e67c3a87f6bcc --- M .travis.yml 1 file changed, 0 insertions(+), 4 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/.travis.yml b/.travis.yml index a1adf16..4759d96 100644 --- a/.travis.yml +++ b/.travis.yml @@ -45,10 +45,6 @@ - LANGUAGE=test FAMILY=wikidata - LANGUAGE=fr FAMILY=wiktionary -matrix: - allow_failures: - - python: "2.6" - notifications: irc: channels: -- To view, visit https://gerrit.wikimedia.org/r/154480 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: If3f6414f48a10b995b43b9e2826e67c3a87f6bcc Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] timestripper: match long names first for months - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: timestripper: match long names first for months ...................................................................... timestripper: match long names first for months Month names sorted in length to match 'long' names before 'short' ones. Change-Id: I62bc3099b97bbc1e3a04e7da1fc5240f175f356e --- M pywikibot/textlib.py M tests/timestripper_tests.py 2 files changed, 9 insertions(+), 7 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/textlib.py b/pywikibot/textlib.py index 03c2d0a..d7f1f87 100644 --- a/pywikibot/textlib.py +++ b/pywikibot/textlib.py @@ -1196,7 +1196,9 @@ # removed; will be handled as digits in regex, adding d+{1,2}\.? escaped_months = [_ for _ in self.origNames2monthNum if not _.strip('.').isdigit()] - escaped_months = [re.escape(_) for _ in escaped_months] + # match longest names first. + escaped_months = [re.escape(_) for + _ in sorted(escaped_months, reverse=True)] # work around for cs wiki: if month are in digits, we assume # that format is dd. mm. (with dot and spaces optional) diff --git a/tests/timestripper_tests.py b/tests/timestripper_tests.py index 9d12b04..c66e69f 100644 --- a/tests/timestripper_tests.py +++ b/tests/timestripper_tests.py @@ -52,23 +52,23 @@ None) ) - txtWithOneMatch = u'this string has XXX, YYY and fév in it' - txtWithTwoMatch = u'this string has XXX, mars and fév in it' - txtWithThreeMatch = u'this string has avr, mars and fév in it' + txtWithOneMatch = u'this string has XXX, YYY and février in it' + txtWithTwoMatch = u'this string has XXX, mars and février in it' + txtWithThreeMatch = u'this string has avr, mars and février in it' txtWithNoMatch = u'this string has no match' pat = self.ts.pmonthR self.assertEqual(self.ts.last_match_and_replace(txtWithOneMatch, pat), (u'this string has XXX, YYY and @@ in it', - {'month': u'fév'}) + {'month': u'février'}) ) self.assertEqual(self.ts.last_match_and_replace(txtWithTwoMatch, pat), (u'this string has XXX, @@ and @@ in it', - {'month': u'fév'}) + {'month': u'février'}) ) self.assertEqual(self.ts.last_match_and_replace(txtWithThreeMatch, pat), (u'this string has @@, @@ and @@ in it', - {'month': u'fév'}) + {'month': u'février'}) ) self.assertEqual(self.ts.last_match_and_replace(txtWithNoMatch, pat), (txtWithNoMatch, -- To view, visit https://gerrit.wikimedia.org/r/154474 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I62bc3099b97bbc1e3a04e7da1fc5240f175f356e Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] Make sure unittest2 is only used on Python 2.6 - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Make sure unittest2 is only used on Python 2.6 ...................................................................... Make sure unittest2 is only used on Python 2.6 Change-Id: I4fe7e3497d91bf3b5d53d649040490d02bc5613b --- M tests/utils.py 1 file changed, 3 insertions(+), 8 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/tests/utils.py b/tests/utils.py index f767538..ddaaf9e 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -9,16 +9,11 @@ # import time import sys -try: + +if sys.version_info < (2, 7): # Unittest2 is a backport of python 2.7s unittest module to python 2.6 - # Trying to import unittest2 has to happen first because 2.6 does have a - # unittest module in the standard library but that doesn't support all the - # features of the one found in python 2.7, so importing unittest first and - # then trying to figure out if it supports the features used would mean - # checking the module contents etc. Just catching an ImportError once is - # much easier. import unittest2 as unittest -except ImportError: +else: import unittest import pywikibot -- To view, visit https://gerrit.wikimedia.org/r/154478 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I4fe7e3497d91bf3b5d53d649040490d02bc5613b Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] Monkey patch unittest -> unittest2 - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Monkey patch unittest -> unittest2 ...................................................................... Monkey patch unittest -> unittest2 This should fix all mixups between unittest and unittest2 in setup.py runs. In addition, both Python 2.6 /and/ 2.7 will now consistently use unittest2 on Travis. Change-Id: I5c9b380e01a53b4c96ca8126fca55bfbc482dcdf --- M .travis.yml M setup.py 2 files changed, 22 insertions(+), 14 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/.travis.yml b/.travis.yml index ac1c930..a1adf16 100644 --- a/.travis.yml +++ b/.travis.yml @@ -27,6 +27,8 @@ - python setup.py install - cd ../.. + - pip install unittest2 + script: - python setup.py test diff --git a/setup.py b/setup.py index e97d637..b4dc10f 100644 --- a/setup.py +++ b/setup.py @@ -14,12 +14,6 @@ import os import itertools -from ez_setup import use_setuptools -use_setuptools() - -from setuptools import setup, find_packages -from setuptools.command import install - test_deps = [] testcollector = "tests" @@ -39,7 +33,12 @@ if sys.version_info < (2, 6, 5): raise RuntimeError("ERROR: Pywikibot only runs under Python 2.6.5 or higher") elif sys.version_info[1] == 6: - test_deps.append('unittest2') + # work around distutils hardcoded unittest dependency + import unittest + if 'test' in sys.argv and sys.version_info < (2, 7): + import unittest2 + sys.modules['unittest'] = unittest2 + script_deps['replicate_wiki.py'] = ['argparse'] testcollector = "tests.utils.collector" dependencies.append('ordereddict') @@ -58,6 +57,20 @@ # when trying to build the C modules. dependencies.append('mwparserfromhell>=0.3.3') +extra_deps.update(script_deps) +# Add script dependencies as test dependencies, +# so scripts can be compiled in test suite. +if 'PYSETUP_TEST_EXTRAS' in os.environ: + test_deps += list(itertools.chain(*(script_deps.values()))) + + +# late import of setuptools due to monkey-patching above +from ez_setup import use_setuptools +use_setuptools() + +from setuptools import setup, find_packages +from setuptools.command import install + class pwb_install(install.install): @@ -73,13 +86,6 @@ python = sys.executable python = python.replace("pythonw.exe", "python.exe") # for Windows subprocess.call([python, "generate_user_files.py"]) - - -extra_deps.update(script_deps) -# Add script dependencies as test dependencies, -# so scripts can be compiled in test suite. -if 'PYSETUP_TEST_EXTRAS' in os.environ: - test_deps += list(itertools.chain(*(script_deps.values()))) setup( name='pywikibot', -- To view, visit https://gerrit.wikimedia.org/r/154465 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5c9b380e01a53b4c96ca8126fca55bfbc482dcdf Gerrit-PatchSet: 8 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] Mark additional tests as unconnected to a Site - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Mark additional tests as unconnected to a Site ...................................................................... Mark additional tests as unconnected to a Site * Namespace tests do not need a site. * One of the textlib tests does not need a site. * One of the wikidataquery tests talks to wikidataquery but does not need a site. Change-Id: Iffacec799d421a678c45ea82e1e32abf400e34fd --- M tests/namespace_tests.py M tests/textlib_tests.py M tests/wikidataquery_tests.py 3 files changed, 8 insertions(+), 7 deletions(-) Approvals: Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/tests/namespace_tests.py b/tests/namespace_tests.py index b6fe44f..70ff073 100644 --- a/tests/namespace_tests.py +++ b/tests/namespace_tests.py @@ -11,14 +11,14 @@ from collections import Iterable from pywikibot.site import Namespace -from tests.utils import PywikibotTestCase, unittest +from tests.utils import NoSiteTestCase, unittest import sys if sys.version_info[0] > 2: basestring = (str, ) -class TestNamespaceObject(PywikibotTestCase): +class TestNamespaceObject(NoSiteTestCase): """Test cases for Namespace class.""" # These should work in any MW wiki diff --git a/tests/textlib_tests.py b/tests/textlib_tests.py index edcead9..b57d979 100644 --- a/tests/textlib_tests.py +++ b/tests/textlib_tests.py @@ -17,7 +17,7 @@ import pywikibot.textlib as textlib from pywikibot import config -from tests.utils import unittest, PywikibotTestCase +from tests.utils import unittest, NoSiteTestCase, PywikibotTestCase files = {} dirname = os.path.join(os.path.dirname(__file__), "pages") @@ -27,9 +27,8 @@ 'r', 'utf-8').read() -class TestSectionFunctions(PywikibotTestCase): +class TestSectionFunctions(NoSiteTestCase): def setUp(self): - self.site = pywikibot.Site('en', 'wikipedia') self.catresult1 = ('[[Category:Cat1]]%(LS)s[[Category:Cat2]]%(LS)s' % {'LS': config.LS}) super(TestSectionFunctions, self).setUp() diff --git a/tests/wikidataquery_tests.py b/tests/wikidataquery_tests.py index f1044b8..ae6a08f 100644 --- a/tests/wikidataquery_tests.py +++ b/tests/wikidataquery_tests.py @@ -12,7 +12,7 @@ import pywikibot.data.wikidataquery as query -from tests.utils import unittest, PywikibotTestCase +from tests.utils import unittest, NoSiteTestCase, PywikibotTestCase import pywikibot from pywikibot.page import ItemPage, PropertyPage, Claim @@ -228,7 +228,9 @@ self.assertEqual(qs, "q=link%5Benwiki%5D&labels=en,fr&props=prop") -class TestApiSlowFunctions(PywikibotTestCase): +class TestApiSlowFunctions(NoSiteTestCase): + + net = True def testQueryApiGetter(self): """ -- To view, visit https://gerrit.wikimedia.org/r/154392 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iffacec799d421a678c45ea82e1e32abf400e34fd Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] Make protect.py more wiki-agnostic - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Make protect.py more wiki-agnostic ...................................................................... Make protect.py more wiki-agnostic - Removed most harcoded protection level and types - Applies protection levels only if the make sense depending on Page.applicable_protections(). - The ProtectionRobot class extends pywikibot.Bot - 'protections' dict replaces 'edit', 'move' and 'create' - the 'always' switch is now a standard option - Clarified "-unprotect" option and added "-default[:<level>]" If a protection type (e.g. "edit") is not defined, it chooses by default "sysop". "-default" overwrites this and "-unprotect" acts like "-default:none". Only "-default" will change only the explicitily set protection types. - Fixed in Page.protect() that prompt only defaults to True if protections was None ("deprecated mode") Bug: 55057 - implicitly enable the 'upload'-type protection Change-Id: I41146fa0c161a242834ab0b1fe2954c7b89d7fd1 --- M pywikibot/page.py M scripts/protect.py 2 files changed, 152 insertions(+), 109 deletions(-) Approvals: John Vandenberg: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page.py b/pywikibot/page.py index 969d02b..7108a2b 100644 --- a/pywikibot/page.py +++ b/pywikibot/page.py @@ -1583,7 +1583,7 @@ @deprecate_arg("throttle", None) def protect(self, edit=False, move=False, create=None, upload=None, - unprotect=False, reason=None, prompt=True, protections=None, + unprotect=False, reason=None, prompt=None, protections=None, **kwargs): """(Un)protect a wiki page. Requires administrator status. @@ -1596,7 +1596,8 @@ @type protections: dict @param reason: Reason for the action @type reason: basestring - @param prompt: Whether to ask user for confirmation + @param prompt: Whether to ask user for confirmation (deprecated). + Defaults to protections is None @type prompt: bool """ def deprecated(value, arg_name): @@ -1637,6 +1638,8 @@ protections = dict( [(p_type, "") for p_type in self.applicable_protections()]) answer = 'y' + if called_using_deprecated_arg and prompt is None: + prompt = True if prompt: pywikibot.bot.warning(u'"prompt" argument of protect() is ' 'deprecated') diff --git a/scripts/protect.py b/scripts/protect.py index d7a55b1..256d76b 100644 --- a/scripts/protect.py +++ b/scripts/protect.py @@ -1,41 +1,43 @@ # -*- coding: utf-8 -*- """ This script can be used to protect and unprotect pages en masse. + Of course, you will need an admin account on the relevant wiki. - - These command line parameters can be used to specify which pages to work on: &params; Furthermore, the following command line parameters are supported: --always: Don't prompt to protect pages, just do it. +-always Don't prompt to protect pages, just do it. --summary: Supply a custom edit summary. +-summary: Supply a custom edit summary. Tries to generate summary from + the page selector. If no summary is supplied or couldn't + determine one from the selector it'll ask for one. --unprotect: Actually unprotect pages instead of protecting +-unprotect Acts like "default:none" --edit:PROTECTION_LEVEL Set edit protection level to PROTECTION_LEVEL +-default: Sets the default protection level (default 'sysop'). If no + level is defined it doesn't change unspecified levels. --move:PROTECTION_LEVEL Set move protection level to PROTECTION_LEVEL +-[type]:[level] Set [type] protection level to [level] -## Without support ## -## -create:PROTECTION_LEVEL Set move protection level to PROTECTION_LEVEL ## +Usual values for [level] are: sysop, autoconfirmed, none; further levels may be +provided by some wikis. -Values for PROTECTION_LEVEL are: sysop, autoconfirmed, none. -If an operation parameter (edit, move or create) is not specified, default -protection level is 'sysop' (or 'none' if -unprotect). +For all protection types (edit, move, etc.) it chooses the default protection +level. This is "sysop" or "none" if -unprotect was selected. If multiple +-unprotect or -default are used, only the last occurence is applied. Usage: python protect.py <OPTIONS> Examples: -Protect everything in the category "To protect" prompting. - python protect.py -cat:"To protect" -always +Protect everything in the category 'To protect' prompting. + python protect.py -cat:'To protect' -Unprotect all pages listed in text file "unprotect.txt" without prompting. - python protect.py -file:unprotect.txt -unprotect +Unprotect all pages listed in text file 'unprotect.txt' without prompting. + python protect.py -file:unprotect.txt -unprotect -always """ # @@ -50,8 +52,7 @@ # import pywikibot -from pywikibot import i18n -from pywikibot import pagegenerators +from pywikibot import i18n, pagegenerators, Bot # This is required for the text that is shown when you run this script # with the parameter -help. @@ -60,141 +61,180 @@ } -class ProtectionRobot: +class ProtectionRobot(Bot): + """ This bot allows protection of pages en masse. """ - def __init__(self, generator, summary, always=False, unprotect=False, - edit='sysop', move='sysop', create='sysop'): + def __init__(self, generator, protections, **kwargs): """ - Arguments: - * generator - A page generator. - * always - Protect without prompting? - * edit, move, create - protection level for these operations - * unprotect - unprotect pages (and ignore edit, move, create params) + Create a new ProtectionRobot. + @param generator: the page generator + @type generator: generator + @param protections: protections as a dict with "type": "level" + @type protections: dict + @param kwargs: additional arguments directly feed to Bot.__init__() """ + self.availableOptions.update({ + 'summary': None, + }) + super(ProtectionRobot, self).__init__(**kwargs) self.generator = generator - self.summary = summary - self.prompt = not always - self.unprotect = unprotect - self.edit = edit - self.move = move + self.protections = protections def run(self): - """ Start the bot's action. - Loop through everything in the page generator and (un)protect it. + """Start the bot's action. + Loop through everything in the page generator and apply the + protections. """ for page in self.generator: pywikibot.output(u'Processing page %s' % page.title()) - page.protect(unprotect=self.unprotect, reason=self.summary, - prompt=self.prompt, edit=self.edit, move=self.move) + if not self.getOption('always'): + choice = pywikibot.inputChoice( + u'Do you want to change the protection level of %s?' + % page.title(asLink=True, forceInterwiki=True), + ['yes', 'No', 'all'], + ['y', 'N', 'a'], + 'n') + if choice == 'n': + continue + elif choice == 'a': + self.option['always'] = True + applicable = page.applicable_protections() + protections = dict( + [prot for prot in self.protections if prot[0] in applicable]) + page.protect(reason=self.getOption('summary'), + protections=protections) -def choiceProtectionLevel(operation, default, protectionLevels): - """ Asks a valid protection level for "operation". - Returns the protection level chosen by user. +def check_protection_level(operation, level, levels, default=None): + """Check if the protection level is valid or asks if necessary. + @return a valid protection level + @rtype string """ - default = default[0] - firstChar = map(lambda level: level[0], protectionLevels) - choiceChar = pywikibot.inputChoice('Choice a protection level to %s:' - % operation, - protectionLevels, firstChar, - default=default) - for level in protectionLevels: - if level.startswith(choiceChar): - return level + if level not in levels: + first_char = [] + default_char = None + num = 1 + for level in levels: + for c in level: + if c not in first_char: + first_char.append(c) + break + else: + first_char.append(unicode(num)) + num += 1 + if level == default: + default_char = first_char[-1] + choice = pywikibot.inputChoice('Choice a protection level to %s:' + % operation, levels, first_char, + default=default_char) + + return levels[first_char.index(choice)] + else: + return level def main(*args): - protectionLevels = ['sysop', 'autoconfirmed', 'none'] - - # This factory is responsible for processing command line arguments - # that are also used by other scripts and that determine on which pages - # to work on. - pageName = '' - summary = None - always = False + options = {} + message_properties = {} generator = None - edit = '' - move = '' - defaultProtection = 'sysop' + protections = {} + default_level = 'sysop' + default_summaries = { + 'cat': 'category', + 'links': 'links', + 'ref': 'ref', + 'imageused': 'images', + 'file': 'simple', + } # read command line parameters local_args = pywikibot.handleArgs(*args) genFactory = pagegenerators.GeneratorFactory() - mysite = pywikibot.Site() + site = pywikibot.Site() + protection_levels = set(site.protection_levels()) + protection_types = site.protection_types() + if '' in protection_levels: + protection_levels.remove('') + protection_levels.add('none') for arg in local_args: if arg == '-always': - always = True + options['always'] = True elif arg.startswith('-summary'): if len(arg) == len('-summary'): - summary = pywikibot.input(u'Enter a reason for the protection:') + # fill dummy value to prevent automatic generation + options['summary'] = None else: - summary = arg[len('-summary:'):] + options['summary'] = arg[len('-summary:'):] elif arg.startswith('-images'): pywikibot.output('\n\03{lightred}-image option is deprecated. ' 'Please use -imagelinks instead.\03{default}\n') local_args.append('-imagelinks' + arg[7:]) elif arg.startswith('-unprotect'): - defaultProtection = 'none' - elif arg.startswith('-edit'): - edit = arg[len('-edit:'):] - if edit not in protectionLevels: - edit = choiceProtectionLevel( - 'edit', defaultProtection, protectionLevels) - elif arg.startswith('-move'): - move = arg[len('-move:'):] - if move not in protectionLevels: - move = choiceProtectionLevel( - 'move', defaultProtection, protectionLevels) - elif arg.startswith('-create'): - create = arg[len('-create:'):] - if create not in protectionLevels: - create = choiceProtectionLevel( - 'create', defaultProtection, protectionLevels) + default_level = 'none' + elif arg.startswith('-default'): + if len(arg) == len('-default'): + default_level = None + else: + default_level = arg[len('-default:'):] else: - genFactory.handleArg(arg) - found = arg.find(':') + 1 - if found: - pageName = arg[found:] + is_p_type = False + if arg.startswith('-'): + delimiter = arg.find(':') + if delimiter > 0: + p_type = arg[1:delimiter] + level = arg[delimiter + 1:] + if p_type in protection_types: + protections[p_type] = level + is_p_type = True + if not is_p_type: + if not genFactory.handleArg(arg): + raise ValueError('Unknown parameter "{}"'.format(arg)) + found = arg.find(':') + 1 + if found: + message_properties.update({'cat': arg[found:], + 'page': arg[found:]}) - if not summary: - if pageName: - if arg.startswith('cat') or arg.startswith('subcats'): - summary = i18n.twtranslate(mysite, 'protect-category', - {'cat': pageName}) - elif arg.startswith('links'): - summary = i18n.twtranslate(mysite, 'protect-links', - {'page': pageName}) - elif arg.startswith('ref'): - summary = i18n.twtranslate(mysite, 'protect-ref', - {'page': pageName}) - elif arg.startswith('imageused'): - summary = i18n.twtranslate(mysite, 'protect-images', - {'page': pageName}) - elif arg.startswith('file'): - summary = i18n.twtranslate(mysite, 'protect-simple') + if 'summary' not in options: + generator_type = arg[1:found] if found > 0 else arg[1:] + if generator_type in default_summaries: + message_type = default_summaries[generator_type] + if message_type == 'simple' or message_properties: + options['summary'] = i18n.twtranslate( + site, 'protect-{}'.format(message_type), + message_properties) generator = genFactory.getCombinedGenerator() # We are just protecting pages, so we have no need of using a preloading # page generator to actually get the text of those pages. if generator: - if summary is None: - summary = pywikibot.input(u'Enter a reason for the %sprotection:' - % ['', 'un'][protectionLevels == 'none']) - if not edit: - edit = defaultProtection - if not move: - move = defaultProtection - bot = ProtectionRobot(generator, summary, always, edit=edit, move=move) + if default_level: + default_level = check_protection_level('Default level', + default_level, + protection_levels) + # set the default value for all + # None (not the string 'none') will be ignored by Site.protect() + combined_protections = dict([ + (p_type, default_level) for p_type in protection_types]) + for p_type, level in protections.items(): + level = check_protection_level(p_type, level, protection_levels, + default_level) + if level == 'none': + level = '' + combined_protections[p_type] = level + if not options.get('summary'): + options['summary'] = pywikibot.input( + u'Enter a reason for the protection change:') + bot = ProtectionRobot(generator, combined_protections, **options) bot.run() else: # Show help text from the top of this file pywikibot.showHelp() -if __name__ == "__main__": +if __name__ == '__main__': main() -- To view, visit https://gerrit.wikimedia.org/r/149501 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I41146fa0c161a242834ab0b1fe2954c7b89d7fd1 Gerrit-PatchSet: 7 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: Ricordisamoa <ricordisamoa(a)openmailbox.org> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm.wikipedia(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Multichill <maarten(a)mdammers.nl> Gerrit-Reviewer: Ricordisamoa <ricordisamoa(a)openmailbox.org> Gerrit-Reviewer: Russell Blau <russblau(a)imapmail.org> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

[Gerrit] Fix Namespace.__repr__ - change (pywikibot/core)

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. Change subject: Fix Namespace.__repr__ ...................................................................... Fix Namespace.__repr__ The __repr__ in the recently added Namespace class is not able to be used to reconstruct a a Namespace instance, as it pushed an unexploded kwargs into the constructor, resulting in a variable 'kwargs' being stored in Namespace.info. Change-Id: I41404c57e58cf80ea259924c1b07e7a397bf8324 --- M pywikibot/site.py M tests/namespace_tests.py 2 files changed, 35 insertions(+), 3 deletions(-) Approvals: Mpaa: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/site.py b/pywikibot/site.py index 16415fb..597b35d 100644 --- a/pywikibot/site.py +++ b/pywikibot/site.py @@ -332,10 +332,16 @@ def __repr__(self): """Return a reconstructable representation.""" - return '%s(id=%d, custom_name=%r, canonical_name=%r, aliases=%r, ' \ - 'kwargs=%r)' \ + if self.info: + kwargs = ', ' + ', '.join([key + '=' + repr(value) + for (key, value) in + [(key, self.info[key]) + for key in sorted(self.info)]]) + else: + kwargs = '' + return '%s(id=%d, custom_name=%r, canonical_name=%r, aliases=%r%s)' \ % (self.__class__.__name__, self.id, self.custom_name, - self.canonical_name, self.aliases, self.info) + self.canonical_name, self.aliases, kwargs) @staticmethod def builtin_namespaces(use_image_name=False): diff --git a/tests/namespace_tests.py b/tests/namespace_tests.py index b6fe44f..cbf69c9 100644 --- a/tests/namespace_tests.py +++ b/tests/namespace_tests.py @@ -182,6 +182,32 @@ self.assertEquals(Namespace.normalize_name(u'::File:'), False) self.assertEquals(Namespace.normalize_name(u'::File::'), False) + def test_repr(self): + a = Namespace(id=0, canonical_name=u'Foo') + s = repr(a) + r = "Namespace(id=0, custom_name=u'Foo', canonical_name=u'Foo', aliases=[])" + self.assertEquals(s, r) + + a.info['defaultcontentmodel'] = 'bar' + r = {'defaultcontentmodel': 'bar'} + self.assertEquals(a.info, r) + + s = repr(a) + r = "Namespace(id=0, custom_name=u'Foo', canonical_name=u'Foo', aliases=[], defaultcontentmodel='bar')" + self.assertEquals(s, r) + + a.info['case'] = 'upper' + r = {'defaultcontentmodel': 'bar', 'case': 'upper'} + self.assertEquals(a.info, r) + + s = repr(a) + r = "Namespace(id=0, custom_name=u'Foo', canonical_name=u'Foo', aliases=[], case='upper', defaultcontentmodel='bar')" + self.assertEquals(s, r) + + b = eval(repr(a)) + self.assertEquals(a, b) + self.assertEquals(a.info, b.info) + if __name__ == '__main__': try: -- To view, visit https://gerrit.wikimedia.org/r/153790 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I41404c57e58cf80ea259924c1b07e7a397bf8324 Gerrit-PatchSet: 1 Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Owner: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Ladsgroup <ladsgroup(a)gmail.com> Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: jenkins-bot <>

9 years, 8 months

← Newer
1
...
4
5
6
7
8
9
10
...
20
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits August 2014