pywikibot

pywikibot@lists.wikimedia.org

3 participants
6830 discussions

[Pywikipedia-l] SVN: [4269] trunk/pywikipedia/weblinkchecker.py
by wikipedian＠svn.wikimedia.org 13 Sep '07

13 Sep '07

Revision: 4269 Author: wikipedian Date: 2007-09-13 13:40:12 +0000 (Thu, 13 Sep 2007) Log Message: ----------- bugfix Modified Paths: -------------- trunk/pywikipedia/weblinkchecker.py Modified: trunk/pywikipedia/weblinkchecker.py =================================================================== --- trunk/pywikipedia/weblinkchecker.py 2007-09-13 13:24:42 UTC (rev 4268) +++ trunk/pywikipedia/weblinkchecker.py 2007-09-13 13:40:12 UTC (rev 4269) @@ -173,6 +173,7 @@ elif onlyBracketed: regex = r'\[' + regex linkR = re.compile(regex) + return linkR def weblinksIn(text, withoutBracketed = False, onlyBracketed = False): text = wikipedia.removeDisabledParts(text)

1 0

[Pywikipedia-l] SVN: [4268] trunk/pywikipedia/weblinkchecker.py
by wikipedian＠svn.wikimedia.org 13 Sep '07

13 Sep '07

Revision: 4268 Author: wikipedian Date: 2007-09-13 13:24:42 +0000 (Thu, 13 Sep 2007) Log Message: ----------- extracted method used removeDisabledParts() instead of code duplication Modified Paths: -------------- trunk/pywikipedia/weblinkchecker.py Modified: trunk/pywikipedia/weblinkchecker.py =================================================================== --- trunk/pywikipedia/weblinkchecker.py 2007-09-13 10:20:55 UTC (rev 4267) +++ trunk/pywikipedia/weblinkchecker.py 2007-09-13 13:24:42 UTC (rev 4268) @@ -149,17 +149,7 @@ re.compile('.*[\./(a)]bodo.kommune.no(/.*)?'), # bot can't handle their redirects ] -def weblinksIn(text, withoutBracketed = False, onlyBracketed = False): - text = wikipedia.removeDisabledParts(text) - - # MediaWiki parses templates before parsing external links. Thus, there - # might be a | or a } directly after a URL which does not belong to - # the URL itself. - # Blow up templates with spaces to avoid these problems. - templateWithParamsR = re.compile(r'{{([^}]*?[^ ])\|([^ ][^}]*?)}}', re.DOTALL) - while templateWithParamsR.search(text): - text = templateWithParamsR.sub(r'{{ \1 | \2 }}', text) - +def compileLinkR(withoutBracketed = False, onlyBracketed = False): # RFC 2396 says that URLs may only contain certain characters. # For this regex we also accept non-allowed characters, so that the bot # will later show these links as broken ('Non-ASCII Characters in URL'). @@ -183,9 +173,23 @@ elif onlyBracketed: regex = r'\[' + regex linkR = re.compile(regex) + +def weblinksIn(text, withoutBracketed = False, onlyBracketed = False): + text = wikipedia.removeDisabledParts(text) + + # MediaWiki parses templates before parsing external links. Thus, there + # might be a | or a } directly after a URL which does not belong to + # the URL itself. + # Blow up templates with spaces to avoid these problems. + templateWithParamsR = re.compile(r'{{([^}]*?[^ ])\|([^ ][^}]*?)}}', re.DOTALL) + while templateWithParamsR.search(text): + text = templateWithParamsR.sub(r'{{ \1 | \2 }}', text) + + linkR = compileLinkR(withoutBracketed, onlyBracketed) + # Remove HTML comments in URLs as well as URLs in HTML comments. - # Also remove text inside nowiki links - text = re.sub('(?s)<nowiki>.*?</nowiki>|', '', text) + # Also remove text inside nowiki links etc. + text = wikipedia.removeDisabledParts(text) for m in linkR.finditer(text): yield m.group('url')

1 0

[Pywikipedia-l] [ pywikipediabot-Bugs-1789089 ] UnusedFilesGenerator not working
by SourceForge.net 13 Sep '07

13 Sep '07

Bugs item #1789089, was opened at 2007-09-06 12:29 Message generated for change (Comment added) made by zeroj You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1789089&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Closed Resolution: Accepted Priority: 5 Private: No Submitted By: John Vandenberg (zeroj) Assigned to: Nobody/Anonymous (nobody) Summary: UnusedFilesGenerator not working Initial Comment: When using unusedfiles.py on en.wikisource.org, the UnusedFilesGenerator returns nothing. I presume that this is due to MediaWiki changes in the HTML interface for "Special:Unusedimages". My changes to make my bot functional are found here: http://meta.wikimedia.org/wiki/User:Jayvdb The current patch is attached, and I am happy to work on any suggestions on how this patch can be improved. ---------------------------------------------------------------------- >Comment By: John Vandenberg (zeroj) Date: 2007-09-13 20:27 Message: Logged In: YES user_id=108686 Originator: YES I see you have disabled the Page: check code. That is required on wikisource, which uses the ProofreadPage extension. This is a workaround for a mediawiki bug? When the image is used by a ProofreadPage, it still appears in the mediawiki list of unused images. See http://en.wikisource.org/wiki/User_talk:ThomasV#Proofread_Page_and_Unused_m… ---------------------------------------------------------------------- Comment By: Daniel Herding (wikipedian) Date: 2007-09-13 20:18 Message: Logged In: YES user_id=880694 Originator: NO Thank you. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1789089&group_…

1 0

[Pywikipedia-l] SVN: [4267] trunk/pywikipedia/wikipedia.py
by wikipedian＠svn.wikimedia.org 13 Sep '07

13 Sep '07

Revision: 4267 Author: wikipedian Date: 2007-09-13 10:20:55 +0000 (Thu, 13 Sep 2007) Log Message: ----------- disabled some code because it slows this method down, and because it is unclear what it's supposed to do. Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2007-09-13 10:17:57 UTC (rev 4266) +++ trunk/pywikipedia/wikipedia.py 2007-09-13 10:20:55 UTC (rev 4267) @@ -3813,14 +3813,16 @@ fileext = title[len(title)-3:] if title not in seen and fileext == extension: - # Check whether the media is used in a Proofread page - basename = title[6:] - page = Page(self, 'Page:' + basename) + ## Check whether the media is used in a Proofread page + # code disabled because it slows this method down, and + # because it is unclear what it's supposed to do. + #basename = title[6:] + #page = Page(self, 'Page:' + basename) - if not page.exists(): - seen.add(title) - image = ImagePage(self, title) - yield image + #if not page.exists(): + seen.add(title) + image = ImagePage(self, title) + yield image if not repeat: break

1 0

[Pywikipedia-l] [ pywikipediabot-Bugs-1789089 ] UnusedFilesGenerator not working
by SourceForge.net 13 Sep '07

13 Sep '07

Bugs item #1789089, was opened at 2007-09-06 04:29 Message generated for change (Comment added) made by wikipedian You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1789089&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: John Vandenberg (zeroj) Assigned to: Nobody/Anonymous (nobody) Summary: UnusedFilesGenerator not working Initial Comment: When using unusedfiles.py on en.wikisource.org, the UnusedFilesGenerator returns nothing. I presume that this is due to MediaWiki changes in the HTML interface for "Special:Unusedimages". My changes to make my bot functional are found here: http://meta.wikimedia.org/wiki/User:Jayvdb The current patch is attached, and I am happy to work on any suggestions on how this patch can be improved. ---------------------------------------------------------------------- >Comment By: Daniel Herding (wikipedian) Date: 2007-09-13 12:18 Message: Logged In: YES user_id=880694 Originator: NO Thank you. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1789089&group_…

1 0

[Pywikipedia-l] SVN: [4266] trunk/pywikipedia
by wikipedian＠svn.wikimedia.org 13 Sep '07

13 Sep '07

Revision: 4266 Author: wikipedian Date: 2007-09-13 10:17:57 +0000 (Thu, 13 Sep 2007) Log Message: ----------- applied patch by John Vandenberg - zeroj [ 1789089 ] UnusedFilesGenerator not working Modified Paths: -------------- trunk/pywikipedia/pagegenerators.py trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/pagegenerators.py =================================================================== --- trunk/pywikipedia/pagegenerators.py 2007-09-13 09:34:39 UTC (rev 4265) +++ trunk/pywikipedia/pagegenerators.py 2007-09-13 10:17:57 UTC (rev 4266) @@ -125,10 +125,10 @@ for page in pageWithImages.imagelinks(followRedirects = False, loose = True): yield page -def UnusedFilesGenerator(number = 100, repeat = False, site = None): +def UnusedFilesGenerator(number = 100, repeat = False, site = None, extension = None): if site is None: site = wikipedia.getSite() - for page in site.unusedfiles(number=number, repeat=repeat): + for page in site.unusedfiles(number=number, repeat=repeat, extension=extension): yield wikipedia.ImagePage(page.site(), page.title()) def WithoutInterwikiPageGenerator(number = 100, repeat = False, site = None): Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2007-09-13 09:34:39 UTC (rev 4265) +++ trunk/pywikipedia/wikipedia.py 2007-09-13 10:17:57 UTC (rev 4266) @@ -3797,21 +3797,30 @@ if not repeat: break - def unusedfiles(self, number = 10, repeat = False): + def unusedfiles(self, number = 10, repeat = False, extension = None): throttle = True seen = set() + ns = self.image_namespace() + entryR = re.compile('<a href=".+?" title="(?P<title>%s:.+?)">.+?</a>' % ns) while True: path = self.unusedfiles_address(n=number) get_throttle() html = self.getUrl(path) - entryR = re.compile('<li>\(<a href=".+?" title="(?P<title>.+?)">.+?</a>\) ') for m in entryR.finditer(html): + fileext = None title = m.group('title') + if extension: + fileext = title[len(title)-3:] - if title not in seen: - seen.add(title) - page = ImagePage(self, title) - yield page + if title not in seen and fileext == extension: + # Check whether the media is used in a Proofread page + basename = title[6:] + page = Page(self, 'Page:' + basename) + + if not page.exists(): + seen.add(title) + image = ImagePage(self, title) + yield image if not repeat: break

1 0

[Pywikipedia-l] [ pywikipediabot-Bugs-1792492 ] wrong argument description of pagegenerators.py
by SourceForge.net 13 Sep '07

13 Sep '07

Bugs item #1792492, was opened at 2007-09-11 19:16 Message generated for change (Settings changed) made by wikipedian You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1792492&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Falk Steinhauer (falk_steinhauer) Assigned to: Nobody/Anonymous (nobody) Summary: wrong argument description of pagegenerators.py Initial Comment: I am using snapshot 2007-08-11. There are miscellaneous copyNpaste-errors in string "parameterHelp" of file pagegenerators.py. e.g. -unwatched Work on all articles that are not watched by anyone. Argument can also be given as "-unusedfiles:n" where n is some number (??). must be: -unwatched Work on all articles that are not watched by anyone. Argument can also be given as "-unwatched:n" where n is some number (??). or -start Specifies that the robot should go alphabetically through all pages on the home wiki, starting at the named page. Argument can also be given as "-subcat:pagetitle". must be: -start Specifies that the robot should go alphabetically through all pages on the home wiki, starting at the named page. Argument can also be given as "-start:pagetitle". more wrong descriptions: -weblinks -filelinks ---------------------------------------------------------------------- Comment By: Russell Blau (russblau) Date: 2007-09-11 20:04 Message: Logged In: YES user_id=855050 Originator: NO Fixed in r4243; thanks for the report. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1792492&group_…

1 0

[Pywikipedia-l] SVN: [4265] trunk/pywikipedia/isbn.py
by wikipedian＠svn.wikimedia.org 13 Sep '07

13 Sep '07

Revision: 4265 Author: wikipedian Date: 2007-09-13 09:34:39 +0000 (Thu, 13 Sep 2007) Log Message: ----------- handle ISBN10s with small x as checksum Modified Paths: -------------- trunk/pywikipedia/isbn.py Modified: trunk/pywikipedia/isbn.py =================================================================== --- trunk/pywikipedia/isbn.py 2007-09-13 09:34:00 UTC (rev 4264) +++ trunk/pywikipedia/isbn.py 2007-09-13 09:34:39 UTC (rev 4265) @@ -1275,7 +1275,7 @@ """ result = [] for c in self.code: - if c.isdigit() or c == 'X': + if c.isdigit() or c in 'Xx': result.append(c) elif c != '-': raise InvalidIsbnException('The ISBN %s contains invalid characters.' % self.code) @@ -1295,13 +1295,13 @@ #print checksum lastDigit = self.digits()[-1] #print lastDigit - if not ((checksum == 10 and lastDigit == 'X') or (lastDigit.isdigit() and checksum == int(lastDigit))): + if not ((checksum == 10 and lastDigit in 'Xx') or (lastDigit.isdigit() and checksum == int(lastDigit))): raise InvalidIsbnException('The ISBN checksum of %s is incorrect.' % self.code) def checkValidity(self): if len(self.digits()) != 10: raise InvalidIsbnException('The ISBN %s is not 10 digits long.' % self.code) - if 'X' in self.digits()[:-1]: + if 'X' in self.digits()[:-1] or 'x' in self.digits()[:-1]: raise InvalidIsbnException('ISBN %s: X is only allowed at the end of the ISBN.' % self.code) self.checkChecksum() @@ -1313,11 +1313,18 @@ ISBN number. """ code = '978-' + self.code[:-1] - + #cs = self.calculateChecksum() #code += str(cs) return ISBN13(code, checksumMissing = True) + def format(self): + # load overridden superclass method + ISBN.format(self) + # capitalize checksum + if self.code[-1] == 'x': + self.code = self.code[:-1] + 'X' + def getIsbn(code): try: i = ISBN13(code) @@ -1342,7 +1349,7 @@ return i.code def hyphenateIsbnNumbers(text): - isbnR = re.compile(r'(?<=ISBN )(?P<code>[\d\-]+X?)') + isbnR = re.compile(r'(?<=ISBN )(?P<code>[\d\-]+[Xx]?)') text = isbnR.sub(_hyphenateIsbnNumber, text) return text @@ -1360,7 +1367,7 @@ return i13.code def convertIsbn10toIsbn13(text): - isbnR = re.compile(r'(?<=ISBN )(?P<code>[\d\-]+X?)') + isbnR = re.compile(r'(?<=ISBN )(?P<code>[\d\-]+[Xx]?)') text = isbnR.sub(_isbn10toIsbn13, text) return text @@ -1371,7 +1378,7 @@ self.to13 = to13 self.format = format self.always = always - self.isbnR = re.compile(r'(?<=ISBN )(?P<code>[\d\-]+X?)') + self.isbnR = re.compile(r'(?<=ISBN )(?P<code>[\d\-]+[Xx]?)') def treat(self, page): try:

1 0

[Pywikipedia-l] SVN: [4264] trunk/pywikipedia/fixes.py
by wikipedian＠svn.wikimedia.org 13 Sep '07

13 Sep '07

Revision: 4264 Author: wikipedian Date: 2007-09-13 09:34:00 +0000 (Thu, 13 Sep 2007) Log Message: ----------- improved 'isbn' fix Modified Paths: -------------- trunk/pywikipedia/fixes.py Modified: trunk/pywikipedia/fixes.py =================================================================== --- trunk/pywikipedia/fixes.py 2007-09-13 08:46:41 UTC (rev 4263) +++ trunk/pywikipedia/fixes.py 2007-09-13 09:34:00 UTC (rev 4264) @@ -270,11 +270,14 @@ 'en': u'Robot: Fixing ISBN format', }, 'replacements': [ - (r'ISBN (\d+) (\d+) (\d+) (\d+) (\d)', r'ISBN \1-\2-\3-\4-\5'), # missing hyphens in ISBN13 - (r'ISBN (\d+) (\d+) (\d+) (\d|X|x)', r'ISBN \1-\2-\3-\4'), # missing hyphens in ISBN10 - (ur'ISBN (\d+)–(\d+)–(\d+)–(\d+)–(\d)', r'ISBN \1-\2-\3-\4-\5'), # dashes instead of hyphens in ISBN13 - (ur'ISBN (\d+)–(\d+)–(\d+)–(\d|X|x)', r'ISBN \1-\2-\3-\4'), # dashes instead of hyphens in ISBN10 - (r'ISBN: (\d+)', r'ISBN \1'), # colon + # colon + (r'ISBN: (\d+)', r'ISBN \1'), + # Spaces, dashes, or dots instead of hyphens as separators, + # or spaces between digits and separators. + # Note that these regular expressions also match valid ISBNs, but + # these won't be changed. + (ur'ISBN (\d+) *[\- –\.] *(\d+) *[\- –\.] *(\d+) *[\- –\.] *(\d+) *[\- –\.] *(\d)(?!\d)', r'ISBN \1-\2-\3-\4-\5'), # ISBN13 + (r'ISBN (\d+) *[\- –\.] *(\d+) *[\- –\.] *(\d+) *[\- –\.] *(\d|X|x)(?!\d)', r'ISBN \1-\2-\3-\4'), # ISBN10 ], }, }

1 0

[Pywikipedia-l] SVN: [4263] trunk/pywikipedia/fixes.py
by wikipedian＠svn.wikimedia.org 13 Sep '07

13 Sep '07

Revision: 4263 Author: wikipedian Date: 2007-09-13 08:46:41 +0000 (Thu, 13 Sep 2007) Log Message: ----------- added 'isbn' fix Modified Paths: -------------- trunk/pywikipedia/fixes.py Modified: trunk/pywikipedia/fixes.py =================================================================== --- trunk/pywikipedia/fixes.py 2007-09-13 08:46:31 UTC (rev 4262) +++ trunk/pywikipedia/fixes.py 2007-09-13 08:46:41 UTC (rev 4263) @@ -262,6 +262,19 @@ u'[[17. Juni 1953]]', u'[[11. September 2001]]', ] - }, + 'isbn': { + 'regex': True, + 'msg': { + 'de': u'Bot: Korrigiere ISBN-Format', + 'en': u'Robot: Fixing ISBN format', + }, + 'replacements': [ + (r'ISBN (\d+) (\d+) (\d+) (\d+) (\d)', r'ISBN \1-\2-\3-\4-\5'), # missing hyphens in ISBN13 + (r'ISBN (\d+) (\d+) (\d+) (\d|X|x)', r'ISBN \1-\2-\3-\4'), # missing hyphens in ISBN10 + (ur'ISBN (\d+)–(\d+)–(\d+)–(\d+)–(\d)', r'ISBN \1-\2-\3-\4-\5'), # dashes instead of hyphens in ISBN13 + (ur'ISBN (\d+)–(\d+)–(\d+)–(\d|X|x)', r'ISBN \1-\2-\3-\4'), # dashes instead of hyphens in ISBN10 + (r'ISBN: (\d+)', r'ISBN \1'), # colon + ], + }, }

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot