Pywikipedia-l June 2008

pywikipedia-l@lists.wikimedia.org

32 participants
449 discussions

[ pywikipediabot-Patches-2005964 ] Update/optimization of featured.py
by SourceForge.net 29 Jun '08

29 Jun '08

Patches item #2005964, was opened at 2008-06-29 17:23 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2005964&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Melancholie (melancholie) Assigned to: Nobody/Anonymous (nobody) Summary: Update/optimization of featured.py Initial Comment: 1. added dsb summary 2. removed all namespace prefixes (else they just get obsolete with the time); using English standards in that patch (how to use module dicts?)... Diff patch is attached ;-) See also coloured diff: http://als.wikipedia.org/w/index.php?title=Benutzer%3AMelancholieBot%2Ffeat… ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2005964&group_…

1 0

"RuntimeError: thread already started"
by Chris Watkins 29 Jun '08

29 Jun '08

I'm new to using bots. I've had some success, using it on the wiki appropedia.org, but when I run replace.py without the -always parameter, I get an error. The second time I answer "y" (yes), I get "RuntimeError: thread already started" and the bot stops. E.g. when I run this command: python replace.py -xml:currentdump.xml -excepttext:"[[appropriate technology]]" "appropriate technology" "[[appropriate technology]]" ...the first change is successful, but the next time I hit yes, I get: Do you want to accept these changes? ([y]es, [N]o, [a]ll) y Changing page [[en:Tire shingles]] Traceback (most recent call last): File "replace.py", line 555, in <module> main() File "replace.py", line 551, in main bot.run() File "replace.py", line 325, in run page.put_async(new_text) File "/home/cwg/pwb/wikipedia.py", line 1033, in put_async _putthread.start() File "/usr/lib/python2.5/threading.py", line 434, in start raise RuntimeError("thread already started") RuntimeError: thread already started Any help appreciated! Thanks -- Chris Watkins (a.k.a. Chriswaterguy) Appropedia.org - Sharing knowledge to build rich, sustainable lives. Blog: chriswaterguy.livejournal.com/ Buying at Amazon, eBay etc? Start at http://appropedia.maatiam.com and a percentage of your purchase supports Appropedia - at no extra cost. Where men are the most sure and arrogant, they are commonly the most mistaken, and have there given reins to passion, without that proper deliberation and suspense, which can alone secure them from the grossest absurdities. -- David Hume

2 1

SVN: [5659] trunk/pywikipedia/checkimages.py
by filnik＠svn.wikimedia.org 29 Jun '08

29 Jun '08

Revision: 5659 Author: filnik Date: 2008-06-29 12:54:25 +0000 (Sun, 29 Jun 2008) Log Message: ----------- Fixed self.image that was used in multiple functions causing some nosense. Modified Paths: -------------- trunk/pywikipedia/checkimages.py Modified: trunk/pywikipedia/checkimages.py =================================================================== --- trunk/pywikipedia/checkimages.py 2008-06-29 11:49:33 UTC (rev 5658) +++ trunk/pywikipedia/checkimages.py 2008-06-29 12:54:25 UTC (rev 5659) @@ -465,12 +465,15 @@ self.duplicatesReport = duplicatesReport image_n = self.site.image_namespace() self.image_namespace = "%s:" % image_n # Example: "User_talk:" - def report(self, newtext, image, notification = None, head = None, + def setParameters(self, image): + """ Function to set parameters, now only image but maybe it can be used for others in "future" """ + self.image = image + def report(self, newtext, image_to_report, notification = None, head = None, notification2 = None, unver = True, commTalk = None, commImage = None): """ Function to make the reports easier (or I hope so). """ # Defining some useful variable for next... + self.image_to_report = image_to_report self.newtext = newtext - self.image = image self.head = head self.notification = notification self.notification2 = notification2 @@ -528,7 +531,7 @@ """ Function to add the template in the image and to find out who's the user that has uploaded the image. """ # Defing the image's Page Object - p = wikipedia.ImagePage(self.site, 'Image:%s' % self.image) + p = wikipedia.ImagePage(self.site, 'Image:%s' % self.image_to_report) # Get the image's description try: testoa = p.get() @@ -541,14 +544,14 @@ if put: p.put(testoa + self.newtext, comment = self.commImage, minorEdit = True) # paginetta it's the image page object. - paginetta = wikipedia.ImagePage(self.site, self.image_namespace + self.image) + paginetta = wikipedia.ImagePage(self.site, self.image_namespace + self.image_to_report) try: nick = paginetta.getLatestUploader()[0] except wikipedia.NoPage: - wikipedia.output(u"Seems that %s hasn't the image at all, but there is something in the description..." % self.image) + wikipedia.output(u"Seems that %s hasn't the image at all, but there is something in the description..." % self.image_to_report) repme = "\n*[[:Image:%s]] problems '''with the APIs'''" # We have a problem! Report and exit! - self.report_image(self.image, self.rep_page, self.com, repme) + self.report_image(self.image_to_report, self.rep_page, self.com, repme) return False luser = wikipedia.url2link(nick, self.site, self.site) pagina_discussione = "%s:%s" % (self.site.namespace(3), luser) @@ -565,7 +568,7 @@ if self.notification2 == None: self.notification2 = self.notification else: - self.notification2 = self.notification2 % self.image + self.notification2 = self.notification2 % self.image_to_report second_text = False # Getting the talk page's history, to check if there is another advise... # The try block is used to prevent error if you use an old wikipedia.py's version. @@ -697,9 +700,8 @@ encodedTitle = title.encode(self.site.encoding()) return urllib.quote(encodedTitle) - def checkImageOnCommons(self, image): + def checkImageOnCommons(self): """ Checking if the image is on commons """ - self.image = image wikipedia.output(u'Checking if %s is on commons...' % self.image) commons_site = wikipedia.getSite('commons', 'commons') regexOnCommons = r"\n\*\[\[:Image:%s\]\] is also on '''Commons''': \[\[commons:Image:.*?\]\]$" % self.image @@ -727,7 +729,7 @@ # Problems? No, return True return True - def checkImageDuplicated(self, image): + def checkImageDuplicated(self): """ Function to check the duplicated images. """ # {{Dupe|Image:Blanche_Montel.jpg}} dupText = wikipedia.translate(self.site, duplicatesText) @@ -736,7 +738,6 @@ dupTalkText = wikipedia.translate(self.site, duplicates_user_talk_text) dupComment_talk = wikipedia.translate(self.site, duplicates_comment_talk) dupComment_image = wikipedia.translate(self.site, duplicates_comment_image) - self.image = image duplicateRegex = r'\n\*(?:\[\[:Image:%s\]\] has the following duplicates:|\*\[\[:Image:%s\]\])$' % (self.convert_to_url(self.image), self.convert_to_url(self.image)) imagePage = wikipedia.ImagePage(self.site, 'Image:%s' % self.image) hash_found = imagePage.getHash() @@ -1126,6 +1127,7 @@ except IndexError:# Namespace image not found, that's not an image! Let's skip... wikipedia.output(u"%s is not an image, skipping..." % image.title()) continue + mainClass.setParameters(imageName) # Setting the image for the main class # Skip block if skip == True: # If the images to skip are more the images to check, make them the same number @@ -1164,16 +1166,16 @@ wikipedia.output(u"Skipping %s because it has been deleted." % imageName) continue except wikipedia.IsRedirectPage: - wikipedia.output(u"The file description for %s is a redirect?!" % imageName ) + wikipedia.output(u"The file description for %s is a redirect?!" % imageName) continue # Check on commons if there's already an image with the same name if commonsActive == True: - response = mainClass.checkImageOnCommons(imageName) + response = mainClass.checkImageOnCommons() if response == False: continue # Check if there are duplicates of the image on the project selected if duplicatesActive == True: - response2 = mainClass.checkImageDuplicated(imageName) + response2 = mainClass.checkImageDuplicated() if response2 == False: continue # Is the image already tagged? If yes, no need to double-check, skip

1 0

SVN: [5658] trunk/pywikipedia/noreferences.py
by nicdumz＠svn.wikimedia.org 29 Jun '08

29 Jun '08

Revision: 5658 Author: nicdumz Date: 2008-06-29 11:49:33 +0000 (Sun, 29 Jun 2008) Log Message: ----------- Adding {{muchasref}} Modified Paths: -------------- trunk/pywikipedia/noreferences.py Modified: trunk/pywikipedia/noreferences.py =================================================================== --- trunk/pywikipedia/noreferences.py 2008-06-29 11:24:10 UTC (rev 5657) +++ trunk/pywikipedia/noreferences.py 2008-06-29 11:49:33 UTC (rev 5658) @@ -215,7 +215,7 @@ 'en': [u'Reflist',u'Refs',u'FootnotesSmall',u'Reference', u'Ref-list',u'Reference list',u'References-small',u'Reflink', u'Footnotes',u'FootnotesSmall'], - 'es': ['Listaref', 'Reflist'], + 'es': ['Listaref', 'Reflist', 'muchasref'], 'fi': [u'Viitteet', u'Reflist'], 'fr': [u'Références',u'Notes', u'References', u'Reflist'], 'hu': [u'reflist'],

1 0

SVN: [5657] trunk/pywikipedia/wikipedia.py
by filnik＠svn.wikimedia.org 29 Jun '08

29 Jun '08

Revision: 5657 Author: filnik Date: 2008-06-29 11:24:10 +0000 (Sun, 29 Jun 2008) Log Message: ----------- Little bugfix for image.getHash() Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-06-29 11:16:09 UTC (rev 5656) +++ trunk/pywikipedia/wikipedia.py 2008-06-29 11:24:10 UTC (rev 5657) @@ -2641,25 +2641,29 @@ """ Function that return the Hash of an image in oder to understand if two Images are the same or not. """ - params = { - 'action' :'query', - 'titles' :self.title(), - 'prop' :'imageinfo', - 'iiprop' :'sha1', - } - # First of all we need the Hash that identify an image - data = query.GetData(params, useAPI = True, encodeTitle = False) - pageid = data['query']['pages'].keys()[0] - try: - hash_found = data['query']['pages'][pageid][u'imageinfo'][0][u'sha1'] - except KeyError: - if self.exists(): - raise NoHash('No Hash found in the APIs! Maybe the regex to catch it is wrong or someone has changed the APIs structure.') + if self.exists(): + params = { + 'action' :'query', + 'titles' :self.title(), + 'prop' :'imageinfo', + 'iiprop' :'sha1', + } + # First of all we need the Hash that identify an image + data = query.GetData(params, useAPI = True, encodeTitle = False) + pageid = data['query']['pages'].keys()[0] + try: + hash_found = data['query']['pages'][pageid][u'imageinfo'][0][u'sha1'] + except KeyError: + if self.exists(): + raise NoHash('No Hash found in the APIs! Maybe the regex to catch it is wrong or someone has changed the APIs structure.') + else: + wikipedia.output(u'Image deleted before getting the Hash. Skipping...') + return None else: - wikipedia.output(u'Image deleted before getting the Hash. Skipping...') - return None + return hash_found else: - return hash_found + wikipedia.output(u'Image deleted before getting the Hash. Skipping...') + return None def getFileVersionHistoryTable(self): """Return the version history in the form of a wiki table."""

1 0

SVN: [5656] trunk/pywikipedia/query.py
by filnik＠svn.wikimedia.org 29 Jun '08

29 Jun '08

Revision: 5656 Author: filnik Date: 2008-06-29 11:16:09 +0000 (Sun, 29 Jun 2008) Log Message: ----------- Bugfix, lang -> site Modified Paths: -------------- trunk/pywikipedia/query.py Modified: trunk/pywikipedia/query.py =================================================================== --- trunk/pywikipedia/query.py 2008-06-29 09:54:15 UTC (rev 5655) +++ trunk/pywikipedia/query.py 2008-06-29 11:16:09 UTC (rev 5656) @@ -63,9 +63,9 @@ if verbose: if titlecount > 0: - wikipedia.output(u"Requesting %d titles from %s:%s" % (titlecount, lang, path)) + wikipedia.output(u"Requesting %d titles from %s:%s" % (titlecount, site.lang, path)) else: - wikipedia.output(u"Request %s:%s" % (lang, path)) + wikipedia.output(u"Request %s:%s" % (site.lang, path)) lastError = None retry_idle_time = 5 @@ -81,7 +81,7 @@ except ValueError, error: retryCount -= 1 wikipedia.output(u"Error downloading data: %s" % error) - wikipedia.output(u"Request %s:%s" % (lang, path)) + wikipedia.output(u"Request %s:%s" % (site.lang, path)) wikipedia.debugDump('ApiGetDataParse', site, str(error) + '\n%s' % path, jsontext) lastError = error if retryCount >= 0: @@ -95,7 +95,7 @@ raise lastError -def GetInterwikies( lang, titles, extraParams = None ): +def GetInterwikies(site, titles, extraParams = None ): """ Usage example: data = GetInterwikies('ru','user:yurik') titles may be either ane title (as a string), or a list of strings extraParams if given must be a dict() as taken by GetData() @@ -103,16 +103,16 @@ params = {'titles':ListToParam(titles), 'what' : 'redirects|langlinks'} params = CombineParams( params, extraParams ) - return GetData( lang, params ) + return GetData(site, params ) -def GetLinks( lang, titles, extraParams = None ): +def GetLinks(site, titles, extraParams = None ): """ Get list of templates for the given titles """ params = {'titles':ListToParam(titles), 'what': 'redirects|links'} params = CombineParams( params, extraParams ) - return GetData( lang, params ) + return GetData(site, params ) -def GetDisambigTemplates(lang): +def GetDisambigTemplates(site): """This method will return a set of disambiguation templates. Template:Disambig is always assumed to be default, and will be appended (in localized format) regardless of its existence. @@ -125,7 +125,7 @@ disListName = u"Wikipedia:Disambiguation Templates" disListId = 0 - templateNames = GetLinks(lang, [disListName, disambigName]) + templateNames = GetLinks(site, [disListName, disambigName]) for id, page in templateNames['pages'].iteritems(): if page['title'] == disambigName: if 'normalizedTitle' in page:

1 0

SVN: [5655] trunk/pywikipedia/djvutext.py
by jayvdb＠svn.wikimedia.org 29 Jun '08

29 Jun '08

Revision: 5655 Author: jayvdb Date: 2008-06-29 09:54:15 +0000 (Sun, 29 Jun 2008) Log Message: ----------- Ignore codec issues coming from the djvu text Modified Paths: -------------- trunk/pywikipedia/djvutext.py Modified: trunk/pywikipedia/djvutext.py =================================================================== --- trunk/pywikipedia/djvutext.py 2008-06-29 09:53:10 UTC (rev 5654) +++ trunk/pywikipedia/djvutext.py 2008-06-29 09:54:15 UTC (rev 5655) @@ -119,7 +119,7 @@ cmd = "djvutxt -page=%d \"%s\" \"%s.out\"" % (pageno, self.djvu, self.djvu) os.system ( cmd ) - f = codecs.open("%s.out" % self.djvu, 'r', config.textfile_encoding) + f = codecs.open("%s.out" % self.djvu, 'r', config.textfile_encoding, 'replace') djvutxt = f.read()

1 0

SVN: [5654] trunk/pywikipedia/djvutext.py
by jayvdb＠svn.wikimedia.org 29 Jun '08

29 Jun '08

Revision: 5654 Author: jayvdb Date: 2008-06-29 09:53:10 +0000 (Sun, 29 Jun 2008) Log Message: ----------- Handle localised name for "Page" namespace Modified Paths: -------------- trunk/pywikipedia/djvutext.py Modified: trunk/pywikipedia/djvutext.py =================================================================== --- trunk/pywikipedia/djvutext.py 2008-06-29 08:45:11 UTC (rev 5653) +++ trunk/pywikipedia/djvutext.py 2008-06-29 09:53:10 UTC (rev 5654) @@ -90,6 +90,9 @@ linkingPage = wikipedia.Page(wikipedia.getSite(), self.index) self.prefix = linkingPage.titleWithoutNamespace() + if self.prefix[0:6] == 'Liber:': + self.prefix = self.prefix[6:] + wikipedia.output("Using prefix %s" % self.prefix) gen = self.PagesGenerator() site = wikipedia.getSite() @@ -103,7 +106,9 @@ """ Loads the given page, does some changes, and saves it. """ - page = wikipedia.Page(wikipedia.getSite(), 'Page:%s/%d' % (self.prefix, pageno) ) + site = wikipedia.getSite() + page_namespace = site.family.namespaces[104]['la'] + page = wikipedia.Page(site, '%s:%s/%d' % (page_namespace, self.prefix, pageno) ) exists = page.exists() ################################################################

1 0

SVN: [5653] trunk/pywikipedia/reflinks.py
by nicdumz＠svn.wikimedia.org 29 Jun '08

29 Jun '08

Revision: 5653 Author: nicdumz Date: 2008-06-29 08:45:11 +0000 (Sun, 29 Jun 2008) Log Message: ----------- Doc commit, mostly. (+ one cleanup; + one minor fix when bailing out) Modified Paths: -------------- trunk/pywikipedia/reflinks.py Modified: trunk/pywikipedia/reflinks.py =================================================================== --- trunk/pywikipedia/reflinks.py 2008-06-29 07:46:34 UTC (rev 5652) +++ trunk/pywikipedia/reflinks.py 2008-06-29 08:45:11 UTC (rev 5653) @@ -70,8 +70,11 @@ 'en':u'Bot generated title'} soft404 = re.compile(ur'\D404(\D|\Z)|error|errdoc|Not.{0,3}Found|sitedown|eventlog', re.IGNORECASE) +# matches an URL at the index of a website dirIndex = re.compile(ur'^\w+://[^/]+/((default|index)\.(asp|aspx|cgi|htm|html|phtml|mpx|mspx|php|shtml|var))?$', re.IGNORECASE) +# Extracts the domain name domain = re.compile(ur'^(\w+)://(?:www.|)([^/]+)') + globalbadtitles = """ # is (test| @@ -97,11 +100,13 @@ )\W*$ ) """ +# Language-specific bad titles badtitles = { 'en': '', 'fr': '.*(404|page|site).*en +travaux.*', 'es': '.*sitio.*no +disponible.*' } +# Regex that match bare references linksInRef = re.compile( # bracketed URLs ur'(?i)<ref(?P<name>[^>]*)>\s*\[?(?P<url>(?:http|https|ftp)://(?:' + @@ -109,10 +114,15 @@ ur'^\[\]\s<>"]+$[^\[\]\s<>"]+[^\[\]\s\.:;\\,<>\?"]+|'+ # unbracketed without () ur'[^\[\]\s<>"]+[^\[\]\s$\.:;\\,<>\?"]+|[^\[\]\s<>"]+))[!?,\s]*\]?\s*</ref>') -#http://www.twoevils.org/files/wikipedia/404-links.txt.gz + +# Download this file : +# http://www.twoevils.org/files/wikipedia/404-links.txt.gz +# ( maintained by User:Dispenser ) listof404pages = '404-links.txt' class XmlDumpPageGenerator: + """Xml generator that yiels pages containing bare references""" + def __init__(self, xmlFilename, xmlStart, namespaces): self.xmlFilename = xmlFilename self.xmlStart = xmlStart @@ -145,6 +155,8 @@ return page class RefLink: + """Container to handle a single bare reference""" + def __init__(self, link, name): self.refname = name self.link = link @@ -154,16 +166,20 @@ self.title = None def refTitle(self): + """Returns the <ref> with its new title""" return '<ref%s>[%s %s]</ref>' % (self.refname, self.link, self.title, self.linkComment) def refLink(self): + """No title has been found, return the unbracketed link""" return '<ref%s>%s</ref>' % (self.refname, self.link) def refDead(self): + """Dead link, tag it with a {{dead link}}""" tag = wikipedia.translate(self.site, deadLinkTag) % self.link return '<ref%s>%s</ref>' % (self.refname, tag) def transform(self, ispdf = False): + """Normalize the title""" #convert html entities if not ispdf: self.title = wikipedia.html2unicode(self.title) @@ -185,6 +201,7 @@ #prevent multiple quotes being interpreted as '' or ''' self.title = self.title.replace('\'\'', '\''') self.title = wikipedia.unicode2html(self.title, self.site.encoding()) + # TODO : remove HTML when both opening and closing tags are included def avoid_uppercase(self): """ @@ -207,12 +224,24 @@ class ReferencesRobot: def __init__(self, generator, acceptall = False, limit = None, ignorepdf = False ): + """ + - generator : Page generator + - acceptall : boolean, is -always on ? + - limit : int, stop after n modified pages + - ignorepdf : boolean + """ self.generator = generator self.acceptall = acceptall self.limit = limit self.ignorepdf = ignorepdf self.site = wikipedia.getSite() self.stopPage = wikipedia.translate(self.site, stopPage) + + self.titleBlackList = re.compile( + '(' + globalbadtitles + '|' + wikipedia.translate(self.site, badtitles) + ')', + re.I | re.S | re.X) + self.norefbot = noreferences.NoReferencesBot(None) + try : self.stopPageRevId = wikipedia.Page(self.site, self.stopPage).latestRevision() @@ -220,16 +249,20 @@ wikipedia.output(u'The stop page %s does not exist' % self.stopPage.aslink()) wikipedia.stopme() + sys.exit(1) + + # Regex to grasp content-type meta HTML tag in HTML source self.META_CONTENT = re.compile(ur'(?i)<meta[^>]*content\-type[^>]*>') + # Extract the encoding from a charset property (from content-type !) self.CHARSET = re.compile(ur'(?i)charset\s*=\s*(?P<enc>[^\'";>/]*)') + # Extract html title from page self.TITLE = re.compile(ur'(?is)(?<=<title>).*?(?=</title>)') + # Matches content inside <script>/<style>/HTML comments self.NON_HTML = re.compile(ur'(?is)<script[^>]*>.*?</script>|<style[^>]*>.*?</style>||<!\[CDATA\[.*?\]\]>') + + # Authorized mime types for HTML pages str = ur'application/(?:xhtml\+xml|xml)|text/(?:ht|x)ml' self.MIME = re.compile(str) - self.titleBlackList = re.compile( - '(' + globalbadtitles + '|' + wikipedia.translate(self.site, badtitles) + ')', - re.I | re.S | re.X) - self.norefbot = noreferences.NoReferencesBot(None) def put_page(self, page, new): """ @@ -270,6 +303,10 @@ toStdout = True) def getPDFTitle(self, ref, f): + """ + Use pdfinfo to retrieve title from a PDF. + Unix-only, I'm afraid. + """ wikipedia.output( u'PDF file.' ) fd, infile = tempfile.mkstemp() urlobj = os.fdopen(fd, 'r+w') @@ -293,13 +330,6 @@ urlobj.close() os.unlink(infile) - def replace(self, sub, text, repl): - new_text = text - for match in re.finditer(re.escape(sub), text): - if wikipedia.isDisabled(text, match.start()): - continue - #blablah - def run(self): """ Runs the Bot @@ -361,7 +391,7 @@ repl = ref.refLink() new_text = new_text.replace(match.group(), repl) continue - # Test if the redirect was valid + # Get the real url where we end (http redirects !) redir = f.geturl() if redir != ref.link and domain.findall(redir) == domain.findall(link): if soft404.search(redir) and not soft404.search(ref.link): @@ -383,19 +413,17 @@ wikipedia.output(u'HTTP error (%s) for %s on %s' % (e.code, ref.url, page.aslink()), toStdout = True) - if e.code == 410: # 410 Gone, indicates that the resource has been purposely removed + # 410 Gone, indicates that the resource has been purposely removed + if e.code == 410 or (e.code == 404 and (u'\t%s\t' % ref.url in deadLinks)): repl = ref.refDead() new_text = new_text.replace(match.group(), repl) - elif e.code == 404 and (u'\t%s\t' % ref.url in deadLinks): - repl = ref.refDead() - new_text = new_text.replace(match.group(), repl) continue except (urllib2.URLError, socket.error, IOError, httplib.error), e: #except (urllib2.URLError, socket.timeout, ftplib.error, httplib.error, socket.error), e: - wikipedia.output(u'Can\'t get page %s : %s' % (ref.url, e)) + wikipedia.output(u'Can\'t retrieve page %s : %s' % (ref.url, e)) continue except ValueError: #Known bug of httplib, google for : @@ -412,19 +440,23 @@ enc = [] if meta_content: tag = meta_content.group() + # Prefer the contentType from the HTTP header : if not contentType: contentType = tag s = self.CHARSET.search(tag) if s: tmp = s.group('enc').strip("\"' ").lower() - enc.append(tmp) - if tmp in ("gb 2312", "gb2312", "gb-2312", "gb_2312"): + naked = re.sub('[ _\-]', '', tmp) + # Convert to python correct encoding names + if naked == "gb2312": enc.append("gbk") - if tmp in ("shift jis", "shiftjis", "shift-jis", "shift_jis"): + elif naked == "shiftjis": enc.append("shift jis 2004") enc.append("cp932") - if tmp in ("x euc jp", "x-euc-jp"): + elif naked == "xeucjp": enc.append("euc-jp") + else: + enc.append(tmp) if not contentType: wikipedia.output(u'No content-type found for %s' % ref.link) continue @@ -434,6 +466,8 @@ new_text = new_text.replace(match.group(), repl) continue + # Ugly hacks to try to survive when both server and page return no encoding. + # Uses most used encodings for each national suffix if u'.ru' in ref.link or u'.su' in ref.link: # see http://www.sci.aha.ru/ATL/ra13a.htm : no server encoding, no page encoding enc = enc + ['koi8-r', 'windows-1251'] @@ -460,6 +494,7 @@ continue + # Retrieves the first non empty string inside <title> tags for m in self.TITLE.finditer(u.unicode): t = m.group() if t: @@ -474,8 +509,11 @@ wikipedia.output(u'%s : No title found...' % ref.link) continue if enc and u.originalEncoding not in enc: + # BeautifulSoup thinks that the original encoding of our page was not one + # of the encodings we specified. Output a warning. wikipedia.output(u'\03{lightpurple}ENCODING\03{default} : %s (%s)' % (ref.link, ref.title)) + # XXX Ugly hack if u'Ã©' in ref.title: repl = ref.refLink() new_text = new_text.replace(match.group(), repl) @@ -487,6 +525,8 @@ new_text = new_text.replace(match.group(), repl) wikipedia.output(u'\03{lightred}WARNING\03{default} %s : Blacklisted title (%s)' % (ref.link, ref.title)) continue + + # Truncate long titles. 175 is arbitrary if len(ref.title) > 175: ref.title = ref.title[:175] + "..." @@ -498,15 +538,17 @@ % page.aslink()) continue - # Do not add <references/> to templates ! + # Add <references/> when needed, but ignore templates ! if page.namespace != 10: if self.norefbot.lacksReferences(new_text, verbose=False): new_text = self.norefbot.addReferences(new_text) editedpages += 1 self.put_page(page, new_text) + if self.limit and editedpages >= self.limit: wikipedia.output('Edited %s pages, stopping.' % self.limit) return + if editedpages % 20 == 0: wikipedia.output('\03{lightgreen}Checking stop page...\03{default}') actualRev = wikipedia.Page(self.site,

1 0

SVN: [5652] trunk/pywikipedia/reflinks.py
by nicdumz＠svn.wikimedia.org 29 Jun '08

29 Jun '08

Revision: 5652 Author: nicdumz Date: 2008-06-29 07:46:34 +0000 (Sun, 29 Jun 2008) Log Message: ----------- Do not add <references/> in templates Modified Paths: -------------- trunk/pywikipedia/reflinks.py Modified: trunk/pywikipedia/reflinks.py =================================================================== --- trunk/pywikipedia/reflinks.py 2008-06-29 07:33:44 UTC (rev 5651) +++ trunk/pywikipedia/reflinks.py 2008-06-29 07:46:34 UTC (rev 5652) @@ -498,8 +498,10 @@ % page.aslink()) continue - if self.norefbot.lacksReferences(new_text, verbose=False): - new_text = self.norefbot.addReferences(new_text) + # Do not add <references/> to templates ! + if page.namespace != 10: + if self.norefbot.lacksReferences(new_text, verbose=False): + new_text = self.norefbot.addReferences(new_text) editedpages += 1 self.put_page(page, new_text) if self.limit and editedpages >= self.limit:

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Pywikipedia-l June 2008