Pywikipedia-l December 2008

pywikipedia-l@lists.wikimedia.org

23 participants
115 discussions

[ pywikipediabot-Patches-2456633 ] Summary MultiProject Support and other updates
by SourceForge.net 21 Dec '08

21 Dec '08

Patches item #2456633, was opened at 2008-12-22 05:30 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2456633&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Alex S.H. Lin (lin4h) Assigned to: Nobody/Anonymous (nobody) Summary: Summary MultiProject Support and other updates Initial Comment: simple update: checkimages.py:patch info for zh and ja. Other updates: 1. wikipedia.translate Multi-project support: first use in speedy_delete.py, if the array has key "wikipedia" and [default_family], then replace xdict to language only. 2. force put article with sysop account. I add sysop parameter in put.it can save times when user-codes put the article, it cannot put by bot account and retry by sysop. 3. SSL connection for wikimedia projects. by PhiLiP(a)zh.wikipedia, I combile his wikipedia_family code to family.py and add two new parameters in config.py, if SSL_connection on and default_family in available list, it can connect automatilly. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=2456633&group_…

1 0

SVN: [6179] trunk/pywikipedia/wikipedia.py
by purodha＠svn.wikimedia.org 21 Dec '08

21 Dec '08

Revision: 6179 Author: purodha Date: 2008-12-21 19:43:24 +0000 (Sun, 21 Dec 2008) Log Message: ----------- Error corrections. Allow separator before categories and interlanguage links to be real strings other than cr/lf. Note that processing was not altered for the current cr/lf separator, although part of it is not really logical. Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-12-21 14:34:34 UTC (rev 6178) +++ trunk/pywikipedia/wikipedia.py 2008-12-21 19:43:24 UTC (rev 6179) @@ -67,12 +67,18 @@ decodeEsperantoX: decode Esperanto text using the x convention. encodeEsperantoX: convert wikitext to the Esperanto x-encoding. sectionencode: encode text for use as a section title in wiki-links. + findmarker(text, startwith, append): return a string which is not part + of text + expandmarker(text, marker, separator): return marker string expanded + backwards to include separator occurrences plus whitespace Wikitext manipulation functions for interlanguage links: getLanguageLinks(text,xx): extract interlanguage links from text and return in a dict removeLanguageLinks(text): remove all interlanguage links from text + removeLanguageLinksAndSeparator(text, site, marker, separator = ''): + remove language links, whitespace, preceeding separators from text replaceLanguageLinks(oldtext, new): remove the language links and replace them with links from a dict like the one returned by getLanguageLinks @@ -87,6 +93,8 @@ getCategoryLinks(text): return list of Category objects corresponding to links in text removeCategoryLinks(text): remove all category links from text + replaceCategoryLinksAndSeparator(text, site, marker, separator = ''): + remove language links, whitespace, preceeding separators from text replaceCategoryLinks(oldtext,new): replace the category links in oldtext by those in a list of Category objects replaceCategoryInPlace(text,oldcat,newtitle): replace a single link to @@ -1853,19 +1861,13 @@ thistxt = removeDisabledParts(thistxt) # marker for inside templates or parameters - marker = u'@@' - while marker in thistxt: - marker += u'@' + marker = findmarker(thistxt, u'@@', u'@') # marker for links - marker2 = u'##' - while marker2 in thistxt: - marker2 += u'#' + marker2 = findmarker(thistxt, u'##', u'#') # marker for math - marker3 = u'%%' - while marker2 in thistxt: - marker3 += u'%' + marker3 = findmarker(thistxt, u'%%', u'%') result = [] inside = {} @@ -3452,13 +3454,40 @@ For the tags parameter, see removeDisabledParts() above. """ # Find a marker that is not already in the text. - marker = '@@' - while marker in text: - marker += '@' + marker = findmarker(text, '@@', '@') text = text[:index] + marker + text[index:] text = removeDisabledParts(text, tags) return (marker not in text) +def findmarker(text, startwith = u'@', append = u'@'): + # find a string which is not part of text + if len(append) <= 0: + append = u'@' + mymarker = startwith + while mymarker in text: + mymarker += append + return mymarker + +def expandmarker(text, marker = '', separator = ''): + # set to remove any number of separator occurrences plus arbitrary + # whitespace before, after, and between them, + # by allowing to include them into marker. + if separator: + firstinmarker = text.find(marker) + firstinseparator = firstinmarker + lenseparator = len(separator) + striploopcontinue = True + while firstinseparator > 0 and striploopcontinue: + striploopcontinue = False + if (firstinseparator >= lenseparator) and (separator == text[firstinseparator-lenseparator:firstinseparator]): + firstinseparator -= lenseparator + striploopcontinue = True + elif text[firstinseparator-1] < ' ': + firstinseparator -= 1 + striploopcontinue = True + marker = text[firstinseparator:firstinmarker] + marker + return marker + # Part of library dealing with interwiki language links # Note - MediaWiki supports two kinds of interwiki links; interlanguage and @@ -3532,6 +3561,24 @@ ['nowiki', 'comment', 'math', 'pre', 'source'], marker=marker) return text.strip() +def removeLanguageLinksAndSeparator(text, site = None, marker = '', separator = ''): + """Return text with all interlanguage links, plus any preceeding whitespace + and separateor occurrences removed. + + If a link to an unknown language is encountered, a warning is printed. + If a marker is defined, that string is placed at the location of the + last occurence of an interwiki link (at the end if there are no + interwiki links). + + """ + if separator: + mymarker = findmarker(text, u'@L@') + newtext = removeLanguageLinks(text, site, mymarker) + mymarker = expandmarker(newtext, mymarker, separator) + return newtext.replace(mymarker, marker) + else: + return removeLanguageLinks(text, site, marker) + def replaceLanguageLinks(oldtext, new, site = None, addOnly = False): """Replace interlanguage links in the text with a new set of links. @@ -3540,17 +3587,18 @@ function). """ # Find a marker that is not already in the text. - marker = '@@' - while marker in oldtext: - marker += '@' + marker = findmarker( oldtext, u'@@') if site == None: site = getSite() separator = site.family.interwiki_text_separator - s = interwikiFormat(new, insite = site) + cseparator = site.family.category_text_separator + separatorstripped = separator.strip() + cseparatorstripped = cseparator.strip() if addOnly: s2 = oldtext else: - s2 = removeLanguageLinks(oldtext, site = site, marker = marker) + s2 = removeLanguageLinksAndSeparator(oldtext, site = site, marker = marker, separator = separatorstripped) + s = interwikiFormat(new, insite = site) if s: if site.language() in site.family.interwiki_attop: newtext = s + separator + s2.replace(marker,'').strip() @@ -3563,11 +3611,13 @@ firstafter += len(marker) # Is there any text in the 'after' part that means we should keep it after? if "</noinclude>" in s2[firstafter:]: + if separatorstripped: + s = separator + s newtext = s2[:firstafter].replace(marker,'') + s + s2[firstafter:] elif site.language() in site.family.categories_last: cats = getCategoryLinks(s2, site = site) - s2 = removeCategoryLinks(s2.replace(marker,'').strip(), site) + separator + s - newtext = replaceCategoryLinks(s2, cats, site=site) + s2 = removeCategoryLinksAndSeparator(s2.replace(marker,'',cseparatorstripped).strip(), site) + separator + s + newtext = replaceCategoryLinks(s2, cats, site=site, addOnly=True) else: newtext = s2.replace(marker,'').strip() + separator + s else: @@ -3654,7 +3704,7 @@ """Return text with all category links removed. Put the string marker after the last replacement (at the end of the text - if there is no replacement). + if there is no replacement). """ # This regular expression will find every link that is possibly an @@ -3669,6 +3719,22 @@ text = re.sub('\s*%s' % re.escape(marker), '\r\n' + marker, text.strip()) return text.strip() +def removeCategoryLinksAndSeparator(text, site = None, marker = '', separator = ''): + """Return text with all category links, plus any preceeding whitespace + and separateor occurrences removed. + + Put the string marker after the last replacement (at the end of the text + if there is no replacement). + + """ + if separator: + mymarker = findmarker(text, u'@C@') + newtext = removeCategoryLinks(text, site, mymarker) + mymarker = expandmarker(newtext, mymarker, separator) + return newtext.replace(mymarker, marker) + else: + return removeCategoryLinks(text, site, marker) + def replaceCategoryInPlace(oldtext, oldcat, newcat, site=None): """Replace the category oldcat with the category newcat and return the modified text. @@ -3712,40 +3778,41 @@ """ # Find a marker that is not already in the text. - marker = '@@' - while marker in oldtext: - marker += '@' - + marker = findmarker( oldtext, u'@@') if site is None: site = getSite() if site.sitename() == 'wikipedia:de' and "{{Personendaten" in oldtext: raise Error('The PyWikipediaBot is no longer allowed to touch categories on the German Wikipedia on pages that contain the person data template because of the non-standard placement of that template. See http://de.wikipedia.org/wiki/Hilfe_Diskussion:Personendaten/Archiv/bis_2006…') - separator = site.family.category_text_separator - s = categoryFormat(new, insite = site) + iseparator = site.family.interwiki_text_separator + separatorstripped = separator.strip() + iseparatorstripped = iseparator.strip() if addOnly: s2 = oldtext else: - s2 = removeCategoryLinks(oldtext, site = site, marker = marker) - + s2 = removeCategoryLinksAndSeparator(oldtext, site = site, marker = marker, separator = separatorstripped) + s = categoryFormat(new, insite = site) if s: if site.language() in site.family.category_attop: newtext = s + separator + s2 else: # calculate what was after the categories links on the page + firstafter = s2.find(marker) if firstafter < 0: firstafter = len(s2) else: firstafter += len(marker) # Is there any text in the 'after' part that means we should keep it after? if "</noinclude>" in s2[firstafter:]: + if separatorstripped: + s = separator + s newtext = s2[:firstafter].replace(marker,'') + s + s2[firstafter:] elif site.language() in site.family.categories_last: newtext = s2.replace(marker,'').strip() + separator + s else: interwiki = getLanguageLinks(s2) - s2 = removeLanguageLinks(s2.replace(marker,''), site) + separator + s - newtext = replaceLanguageLinks(s2, interwiki, site) + s2 = removeLanguageLinksAndSeparator(s2.replace(marker,''), site, '', iseparatorstripped) + separator + s + newtext = replaceLanguageLinks(s2, interwiki, site = site, addOnly = True) else: newtext = s2.replace(marker,'') return newtext.strip()

1 0

SVN: [6178] trunk/pywikipedia/category.py
by purodha＠svn.wikimedia.org 21 Dec '08

21 Dec '08

Revision: 6178 Author: purodha Date: 2008-12-21 14:34:34 +0000 (Sun, 21 Dec 2008) Log Message: ----------- Allow underlines instead of spaces in category titles on command line, but do not store them in the wiki. Modified Paths: -------------- trunk/pywikipedia/category.py Modified: trunk/pywikipedia/category.py =================================================================== --- trunk/pywikipedia/category.py 2008-12-21 13:21:02 UTC (rev 6177) +++ trunk/pywikipedia/category.py 2008-12-21 14:34:34 UTC (rev 6178) @@ -918,10 +918,10 @@ elif arg == '-rebuild': catDB.rebuild() elif arg.startswith('-from:'): - oldCatTitle = arg[len('-from:'):] + oldCatTitle = arg[len('-from:'):].replace('_', ' ') fromGiven = True elif arg.startswith('-to:'): - newCatTitle = arg[len('-to:'):] + newCatTitle = arg[len('-to:'):].replace('_', ' ') toGiven = True elif arg == '-batch': batchMode = True

1 0

SVN: [6177] trunk/pywikipedia/wikipedia.py
by purodha＠svn.wikimedia.org 21 Dec '08

21 Dec '08

Revision: 6177 Author: purodha Date: 2008-12-21 13:21:02 +0000 (Sun, 21 Dec 2008) Log Message: ----------- typo Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-12-21 13:08:31 UTC (rev 6176) +++ trunk/pywikipedia/wikipedia.py 2008-12-21 13:21:02 UTC (rev 6177) @@ -2315,8 +2315,8 @@ text = self.get(get_redirect = True) output(u'Cannot delete page %s - marking the page for deletion instead:' % self.aslink()) # Note: Parameters to {{delete}}, and their meanings, vary from one Wikipedia to another. - # If you want or need to use them, you must be careful not to brake others. Else don't. - self.put(u'{{delete}}\n%s ~~~~\n----\n\n%s' % (reason, text), comment = reason) + # If you want or need to use them, you must be careful not to break others. Else don't. + self.put(u'{{delete}}\n%s --~~~~\n----\n\n%s' % (reason, text), comment = reason) return else: raise

1 0

SVN: [6176] trunk/pywikipedia/wikipedia.py
by purodha＠svn.wikimedia.org 21 Dec '08

21 Dec '08

Revision: 6176 Author: purodha Date: 2008-12-21 13:08:31 +0000 (Sun, 21 Dec 2008) Log Message: ----------- Parameters to {{delete}}, their meanings, and order, vary from one Wikipedia to another. You cannot use them globally, that never would work everywhere. Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-12-21 09:08:59 UTC (rev 6175) +++ trunk/pywikipedia/wikipedia.py 2008-12-21 13:08:31 UTC (rev 6176) @@ -2314,7 +2314,9 @@ if mark and self.exists(): text = self.get(get_redirect = True) output(u'Cannot delete page %s - marking the page for deletion instead:' % self.aslink()) - self.put(u'{{delete|%s}}\n%s ~~~~\n----\n\n%s' % (reason, reason, text), comment = reason) + # Note: Parameters to {{delete}}, and their meanings, vary from one Wikipedia to another. + # If you want or need to use them, you must be careful not to brake others. Else don't. + self.put(u'{{delete}}\n%s ~~~~\n----\n\n%s' % (reason, text), comment = reason) return else: raise

1 0

SVN: [6175] trunk/pywikipedia/wikipedia.py
by erwin85＠svn.wikimedia.org 21 Dec '08

21 Dec '08

Revision: 6175 Author: erwin85 Date: 2008-12-21 09:08:59 +0000 (Sun, 21 Dec 2008) Log Message: ----------- Undoing revision 6174 for now Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-12-20 19:51:41 UTC (rev 6174) +++ trunk/pywikipedia/wikipedia.py 2008-12-21 09:08:59 UTC (rev 6175) @@ -1534,8 +1534,14 @@ # without any reason! # raise EditConflict(u'Someone deleted the page.') # No raise, simply define these variables and retry: - predata['wpEdittime'] = time.strftime('%Y%m%d%H%M%S', time.gmtime()) - predata['wpStarttime'] = time.strftime('%Y%m%d%H%M%S', time.gmtime()) + if self._editTime: + predata['wpEdittime'] = self._editTime + else: + predata['wpEdittime'] = time.strftime('%Y%m%d%H%M%S', time.gmtime()) + if self._startTime: + predata['wpStarttime'] = self._startTime + else: + predata['wpStarttime'] = time.strftime('%Y%m%d%H%M%S', time.gmtime()) continue if self.site().has_mediawiki_message("viewsource")\ and self.site().mediawiki_message('viewsource') in data:

1 0

SVN: [6174] trunk/pywikipedia/wikipedia.py
by erwin85＠svn.wikimedia.org 20 Dec '08

20 Dec '08

Revision: 6174 Author: erwin85 Date: 2008-12-20 19:51:41 +0000 (Sat, 20 Dec 2008) Log Message: ----------- Set wpEdittime and wpStarttime to now() when MediaWiki returns wpRecreate Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-12-20 19:35:33 UTC (rev 6173) +++ trunk/pywikipedia/wikipedia.py 2008-12-20 19:51:41 UTC (rev 6174) @@ -1534,14 +1534,8 @@ # without any reason! # raise EditConflict(u'Someone deleted the page.') # No raise, simply define these variables and retry: - if self._editTime: - predata['wpEdittime'] = self._editTime - else: - predata['wpEdittime'] = time.strftime('%Y%m%d%H%M%S', time.gmtime()) - if self._startTime: - predata['wpStarttime'] = self._startTime - else: - predata['wpStarttime'] = time.strftime('%Y%m%d%H%M%S', time.gmtime()) + predata['wpEdittime'] = time.strftime('%Y%m%d%H%M%S', time.gmtime()) + predata['wpStarttime'] = time.strftime('%Y%m%d%H%M%S', time.gmtime()) continue if self.site().has_mediawiki_message("viewsource")\ and self.site().mediawiki_message('viewsource') in data:

2 1

SVN: [6173] trunk/pywikipedia
by purodha＠svn.wikimedia.org 20 Dec '08

20 Dec '08

Revision: 6173 Author: purodha Date: 2008-12-20 19:35:33 +0000 (Sat, 20 Dec 2008) Log Message: ----------- namespacenames updated, plus mt, and few corrections. Modified Paths: -------------- trunk/pywikipedia/families/wikipedia_family.py trunk/pywikipedia/family.py Modified: trunk/pywikipedia/families/wikipedia_family.py =================================================================== --- trunk/pywikipedia/families/wikipedia_family.py 2008-12-20 11:48:07 UTC (rev 6172) +++ trunk/pywikipedia/families/wikipedia_family.py 2008-12-20 19:35:33 UTC (rev 6173) @@ -245,7 +245,7 @@ 'mn': u'Wikipedia-н хэлэлцүүлэг', 'mr': u'विकिपीडिया चर्चा', 'ms': u'Perbincangan Wikipedia', - 'mt': u'Wikipedija diskuti', + 'mt': [u'Wikipedija diskussjoni', u'Wikipedija diskuti'], 'myv': u'Википедиясь кортамось', 'mzn': u'بحث Wikipedia', 'nah': [u'Wikipedia tēixnāmiquiliztli', u'Wikipedia Discusión'], Modified: trunk/pywikipedia/family.py =================================================================== --- trunk/pywikipedia/family.py 2008-12-20 11:48:07 UTC (rev 6172) +++ trunk/pywikipedia/family.py 2008-12-20 19:35:33 UTC (rev 6173) @@ -63,6 +63,7 @@ 'bcl': u'Medio', 'be': u'Мультымедыя', 'be-x-old': u'Мэдыя', + 'bm': u'Média', 'bn': u'Média', 'bg': u'Медия', 'bpy': u'মিডিয়া', @@ -107,6 +108,7 @@ 'mk': u'Медија', 'ml': u'മീഡിയ', 'mr': u'मिडिया', + 'mt': u'Midja', 'mzn': u'مدیا', 'nah': u'Mēdiatl', 'new': u'माध्यम', @@ -131,7 +133,7 @@ 'tl': u'Midya', 'tlh': u'Doch', 'tr': u'Medya', - 'ty': u'Médi', + 'ty': u'Média', 'udm': u'Медиа', 'uk': u'Медіа', 'ur': u'زریعہ', @@ -163,6 +165,7 @@ 'bcl': u'Espesyal', 'be': u'Адмысловае', 'be-x-old': u'Спэцыяльныя', + 'bm': u'Spécial', 'bn': u'Spécial', 'bg': u'Специални', 'bn': u'বিশেষ', @@ -243,6 +246,7 @@ 'ml': u'പ്രത്യേകം', 'mr': u'विशेष', 'ms': u'Khas', + 'mt': u'Specjali', 'mzn': u'ویژه', 'nah': [u'Nōncuahquīzqui', u'Especial'], 'nap': [u'Speciàle', u'Speciale'], @@ -406,6 +410,7 @@ 'ml': u'സംവാദം', 'mr': u'चर्चा', 'ms': u'Perbincangan', + 'mt': u'Diskussjoni', 'mzn': u'بحث', 'nah':[u'Tēixnāmiquiliztli', u'Discusión'], 'nap':[u'Chiàcchiera', u'Discussione'], @@ -569,6 +574,7 @@ 'ml': u'ഉപയോക്താവ്', 'mr': u'सदस्य', 'ms': u'Pengguna', + 'mt': u'Utent', 'mzn': u'کاربر', 'nah': [u'Tlatequitiltilīlli', u'Usuario'], 'nap': u'Utente', @@ -732,6 +738,7 @@ 'ml': u'ഉപയോക്താവിന്റെ സംവാദം', 'mr': u'सदस्य चर्चा', 'ms': u'Perbincangan Pengguna', + 'mt': u'Diskussjoni utent', 'mzn': u'بحث کاربر', 'nah':[u'Tlatequitiltilīlli tēixnāmiquiliztli', u'Usuario Discusión'], 'nap':[u'Utente chiàcchiera', u'Discussioni utente'], @@ -1077,6 +1084,7 @@ 'ml': u'ചിത്രത്തിന്റെ സംവാദം', 'mr': u'चित्र चर्चा', 'ms': [u'Perbincangan Fail', u'Perbincangan Imej'], + 'mt': u'Diskussjoni stampa', 'mzn': u'بحث تصویر', 'nah': [u'Īxiptli tēixnāmiquiliztli', u'Imagen Discusión'], 'nap': [u'Fiùra chiàcchiera', u'Discussioni immagine'], @@ -1097,7 +1105,7 @@ 'pt': [u'Ficheiro Discussão', u'Imagem Discussão'], 'qu': u'Rikcha rimanakuy', 'rmy':[u'Chitro vakyarimata', u'Discuţie Imagine'], - 'ro': [u'Discuţie Fisier', u'Discuţie Imagine'], + 'ro': [u'Discutie Fisier', u'Discuţie Fisier', u'Discuţie Imagine'], 'ru': u'Обсуждение изображения', 'sa': u'चित्रसंभाषणं', 'sc': u'Immàgini contièndha', @@ -1152,7 +1160,7 @@ 'crh': [u'MediaViki', u'МедиаВики'], 'fa': u'مدیاویکی', 'fi': u'Järjestelmäviesti', - 'fo': u'MidiaWiki', + 'fo': [u'MediaWiki', u'MidiaWiki'], 'glk': u'مدیاویکی', 'gu': u'મીડિયાવિકિ', 'he': u'מדיה ויקי', @@ -1234,7 +1242,7 @@ 'ff': u'Discussion MediaWiki', 'fi': u'Keskustelu järjestelmäviestistä', 'fiu-vro': u'MediaWiki arotus', - 'fo': [u'MidiaWiki-kjak', u'MidiaWiki kjak'], + 'fo': [u'MediaWiki-kjak', u'MidiaWiki-kjak', u'MidiaWiki kjak'], 'fr': u'Discussion MediaWiki', 'frp': u'Discussion MediaWiki', 'fur': u'Discussion MediaWiki', @@ -1286,6 +1294,7 @@ 'ml': u'മീഡിയവിക്കി സംവാദം', 'mr': u'मिडियाविकी चर्चा', 'ms': u'Perbincangan MediaWiki', + 'mt': u'Diskussjoni MediaWiki', 'mzn': u'بحث مدیاویکی', 'nah':[u'Huiquimedia tēixnāmiquiliztli', u'MediaWiki Discusión'], 'nap':[u'MediaWiki chiàcchiera', u'Discussioni MediaWiki'], @@ -1443,6 +1452,7 @@ 'ml': u'ഫലകം', 'mr': u'साचा', 'ms': u'Templat', + 'mt': u'Mudell', 'mzn': u'الگو', 'nah': [u'Nemachiyōtīlli', u'Plantilla'], 'nap': u'Modello', @@ -1599,6 +1609,7 @@ 'ml': u'ഫലകത്തിന്റെ സംവാദം', 'mr': u'साचा चर्चा', 'ms': u'Perbincangan Templat', + 'mt': u'Diskussjoni mudell', 'mzn': u'بحث الگو', 'nah':[u'Nemachiyōtīlli tēixnāmiquiliztli', u'Plantilla Discusión'], 'nap':[u'Modello chiàcchiera', u'Discussioni template'], @@ -1755,6 +1766,7 @@ 'ml': u'സഹായം', 'mr': u'साहाय्य', 'ms': u'Bantuan', + 'mt': u'Ghajnuna', 'mzn': u'راهنما', 'nah':[u'Tēpalēhuiliztli', u'Ayuda'], 'nap':[u'Ajùto', u'Aiuto'], @@ -1915,6 +1927,7 @@ 'ml': u'സഹായത്തിന്റെ സംവാദം', 'mr': u'साहाय्य चर्चा', 'ms': u'Perbincangan Bantuan', + 'mt': u'Diskussjoni ghajnuna', 'mzn': u'بحث راهنما', 'nah':[u'Tēpalēhuiliztli tēixnāmiquiliztli', u'Ayuda Discusión'], 'nap':[u'Ajùto chiàcchiera', u'Discussioni aiuto'], @@ -2076,6 +2089,7 @@ 'ml': u'വിഭാഗം', 'mr': u'वर्ग', 'ms': u'Kategori', + 'mt': u'Kategorija', 'mzn': u'رده', 'nah':[u'Neneuhcāyōtl', u'Categoría'], 'nap':[u'Categurìa', u'Categoria'], @@ -2233,11 +2247,12 @@ 'lt': u'Kategorijos aptarimas', 'lv': u'Kategorijas diskusija', 'map-bms': u'Dhiskusi Kategori', - 'mg': [u'inika amin\'ny sokajy', u'Discussion Catégorie'], + 'mg': [u'Dinika amini\'ny sokajy', u'Discussion Catégorie'], 'mk': u'Разговор за категорија', 'ml': u'വിഭാഗത്തിന്റെ സംവാദം', 'mr': u'वर्ग चर्चा', 'ms': u'Perbincangan Kategori', + 'mt': u'Diskussjoni kategorija', 'mzn': u'بحث رده', 'nah':[u'Neneuhcāyōtl tēixnāmiquiliztli', u'Categoría Discusión'], 'nap':[u'Categurìa chiàcchiera', u'Discussioni categoria'],

1 0

SVN: [6172] trunk/pywikipedia/wikipedia.py
by purodha＠svn.wikimedia.org 20 Dec '08

20 Dec '08

Revision: 6172 Author: purodha Date: 2008-12-20 11:48:07 +0000 (Sat, 20 Dec 2008) Log Message: ----------- Errors with misplaced Category block and/or interlanguage links under certain conditions fixed, slightly more systematic code. Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-12-19 18:16:50 UTC (rev 6171) +++ trunk/pywikipedia/wikipedia.py 2008-12-20 11:48:07 UTC (rev 6172) @@ -3530,7 +3530,7 @@ ['nowiki', 'comment', 'math', 'pre', 'source'], marker=marker) return text.strip() -def replaceLanguageLinks(oldtext, new, site = None): +def replaceLanguageLinks(oldtext, new, site = None, addOnly = False): """Replace interlanguage links in the text with a new set of links. 'new' should be a dict with the Site objects as keys, and Page objects @@ -3543,15 +3543,22 @@ marker += '@' if site == None: site = getSite() + separator = site.family.interwiki_text_separator s = interwikiFormat(new, insite = site) - s2 = removeLanguageLinks(oldtext, site = site, marker = marker) + if addOnly: + s2 = oldtext + else: + s2 = removeLanguageLinks(oldtext, site = site, marker = marker) if s: - separator = site.family.interwiki_text_separator if site.language() in site.family.interwiki_attop: newtext = s + separator + s2.replace(marker,'').strip() else: # calculate what was after the language links on the page - firstafter = s2.find(marker) + len(marker) + firstafter = s2.find(marker) + if firstafter < 0: + firstafter = len(s2) + else: + firstafter += len(marker) # Is there any text in the 'after' part that means we should keep it after? if "</noinclude>" in s2[firstafter:]: newtext = s2[:firstafter].replace(marker,'') + s + s2[firstafter:] @@ -3712,6 +3719,7 @@ if site.sitename() == 'wikipedia:de' and "{{Personendaten" in oldtext: raise Error('The PyWikipediaBot is no longer allowed to touch categories on the German Wikipedia on pages that contain the person data template because of the non-standard placement of that template. See http://de.wikipedia.org/wiki/Hilfe_Diskussion:Personendaten/Archiv/bis_2006…') + separator = site.family.category_text_separator s = categoryFormat(new, insite = site) if addOnly: s2 = oldtext @@ -3719,12 +3727,14 @@ s2 = removeCategoryLinks(oldtext, site = site, marker = marker) if s: - separator = site.family.category_text_separator if site.language() in site.family.category_attop: newtext = s + separator + s2 else: # calculate what was after the categories links on the page - firstafter = s2.find(marker) + len(marker) + if firstafter < 0: + firstafter = len(s2) + else: + firstafter += len(marker) # Is there any text in the 'after' part that means we should keep it after? if "</noinclude>" in s2[firstafter:]: newtext = s2[:firstafter].replace(marker,'') + s + s2[firstafter:] @@ -3735,8 +3745,7 @@ s2 = removeLanguageLinks(s2.replace(marker,''), site) + separator + s newtext = replaceLanguageLinks(s2, interwiki, site) else: - s2 = s2.replace(marker,'') - return s2 + newtext = s2.replace(marker,'') return newtext.strip() def categoryFormat(categories, insite = None):

1 0

SVN: [6171] branches/rewrite/pywikibot
by russblau＠svn.wikimedia.org 19 Dec '08

19 Dec '08

Revision: 6171 Author: russblau Date: 2008-12-19 18:16:50 +0000 (Fri, 19 Dec 2008) Log Message: ----------- More output formatting, and login bug fixes Modified Paths: -------------- branches/rewrite/pywikibot/__init__.py branches/rewrite/pywikibot/bot.py branches/rewrite/pywikibot/comms/http.py branches/rewrite/pywikibot/data/api.py branches/rewrite/pywikibot/login.py Modified: branches/rewrite/pywikibot/__init__.py =================================================================== --- branches/rewrite/pywikibot/__init__.py 2008-12-19 12:30:48 UTC (rev 6170) +++ branches/rewrite/pywikibot/__init__.py 2008-12-19 18:16:50 UTC (rev 6171) @@ -18,7 +18,9 @@ import textlib from bot import * +logging.basicConfig(fmt="%(message)s") + def deprecate_arg(old_arg, new_arg): """Decorator to declare old_arg deprecated and replace it with new_arg""" logger = logging.getLogger() @@ -102,30 +104,6 @@ link_regex = re.compile(r'\[\[(?P<title>[^\]|[#<>{}]*)(\|.*?)?\]\]') -# User interface functions (kept extremely simple for debugging) - -def output(text, toStdout=False): - if toStdout: - level = STDOUT - else: - level = logging.INFO - logging.getLogger().log(level, - text.encode(config.console_encoding, - "xmlcharrefreplace")) - -def input(prompt, password=False): - if isinstance(prompt, unicode): - encoding = sys.stdout.encoding - # Fallback to _some_ encoding for virtual consoles (cron, screen !) - if not encoding: - encoding = "UTF-8" - prompt = prompt.encode(encoding, "xmlcharrefreplace") - if password: - import getpass - return getpass.getpass(prompt) - return raw_input(prompt) - - def set_debug(layer): """Set the logger for specified layer to DEBUG level. @@ -144,7 +122,7 @@ This method does not check the 'layer' argument for validity. """ - logging.getLogger(layer).setLevel(logging.DEBUG) + logging.getLogger(layer).setLevel(DEBUG) # Throttle and thread handling @@ -175,7 +153,7 @@ # only need one drop() call because all throttles use the same global pid try: _sites[_sites.keys()[0]].throttle.drop() - logger.log(pywikibot.VERBOSE, "Dropped throttle(s).") + logger.log(VERBOSE, "Dropped throttle(s).") except IndexError: pass Modified: branches/rewrite/pywikibot/bot.py =================================================================== --- branches/rewrite/pywikibot/bot.py 2008-12-19 12:30:48 UTC (rev 6170) +++ branches/rewrite/pywikibot/bot.py 2008-12-19 18:16:50 UTC (rev 6171) @@ -23,11 +23,26 @@ # logging levels +from logging import DEBUG, INFO, WARNING, ERROR, CRITICAL STDOUT = 16 VERBOSE = 18 INPUT = 25 +def output(text, decoder=None, newline=True, toStdout=False, level=INFO): + if toStdout: + level = STDOUT + logging.getLogger().log(level, text) + + +def input(prompt, password=False): + logging.getLogger().log(INPUT, prompt) + if password: + import getpass + return getpass.getpass("") + return raw_input() + + def calledModuleName(): """Return the name of the module calling this function. @@ -42,14 +57,19 @@ return os.path.basename(called) -class LevelFilter(logging.Filter): - """Filter that only passes records at a specific level.""" +class MaxLevelFilter(logging.Filter): + """Filter that only passes records at or below a specific level. + + (setting handler level only passes records at or *above* a specified level, + so this provides the opposite functionality) + + """ def __init__(self, level=None): self.level = level def filter(self, record): if self.level: - return record.levelno == self.level + return record.levelno <= self.level else: return True @@ -93,6 +113,7 @@ if not moduleName: moduleName = "terminal-interface" nonGlobalArgs = [] + username = None for arg in args: arg = _decodeArg(arg) if arg == '-help': @@ -101,7 +122,9 @@ elif arg.startswith('-family:'): config.family = arg[8:] elif arg.startswith('-lang:'): - config.code = arg[6:] + config.mylang = arg[6:] + elif arg.startswith("-user:"): + username = arg[6:] elif arg.startswith('-putthrottle:'): config.put_throttle = int(arg[13:]) elif arg.startswith('-pt:'): @@ -133,8 +156,29 @@ # about it. nonGlobalArgs.append(arg) + if username: + config.usernames[config.family][config.mylang] = username + # initialize logging system for terminal-based bots + # All user output is routed through the logging module. + # Each type of output is handled by an appropriate handler object. + # This structure is used to permit eventual development of other + # user interfaces (GUIs) without modifying the core bot code. + # The following output levels are defined: + # DEBUG - only for file logging; debugging messages + # STDOUT - output that must be sent to sys.stdout (for bots that may + # have their output redirected to a file or other destination) + # VERBOSE - optional progress information for display to user + # INFO - normal (non-optional) progress information for display to user + # INPUT - prompts requiring user response + # WARN - user warning messages + # ERROR - user error messages + # CRITICAL - fatal error messages + # Accordingly, do ''not'' use print statements in bot code; instead, + # send output to the pywikibot.output() function which will route it + # to the logging module. + logging.addLevelName(VERBOSE, "VERBOSE") # for messages to be displayed on terminal at "verbose" setting # use INFO for messages to be displayed even on non-verbose setting @@ -143,15 +187,20 @@ logging.addLevelName(INPUT, "INPUT") # for prompts requiring user response - logging.basicConfig(format="%(message)s") # initialize root logger root_logger = logging.getLogger() + # default handler for VERBOSE and INFO levels default_handler = root_logger.handlers[0] - root_logger.setLevel(logging.DEBUG) # all records go to logger - # handlers filter separately by level + root_logger.setLevel(DEBUG) # all records go to logger + + # configure default handler for VERBOSE, INFO, and INPUT levels if config.verbose_output: default_handler.setLevel(VERBOSE) else: - default_handler.setLevel(logging.INFO) + default_handler.setLevel(INFO) + default_handler.addFilter(MaxLevelFilter(INPUT)) + default_handler.setFormatter(logging.Formatter(fmt="%(message)s")) + + # if user has enabled file logging, configure file handler if moduleName in config.log or '*' in config.log: if config.logfilename: logfile = config.datafilepath(config.logfilename) @@ -160,7 +209,7 @@ file_handler = logging.handlers.RotatingFileHandler( filename=logfile, maxBytes=2 << 20, backupCount=5) if config.debug_log: - file_handler.setLevel(logging.DEBUG) + file_handler.setLevel(DEBUG) else: file_handler.setLevel(VERBOSE) form = logging.Formatter( @@ -171,11 +220,19 @@ file_handler.setFormatter(form) root_logger.addHandler(file_handler) + # handler for level STDOUT output_handler = logging.StreamHandler(strm=sys.stdout) output_handler.setLevel(STDOUT) - output_handler.addFilter(LevelFilter(STDOUT)) + output_handler.addFilter(MaxLevelFilter(STDOUT)) root_logger.addHandler(output_handler) + # handler for levels WARNING and higher + warning_handler = logging.StreamHandler() # uses sys.stderr + warning_handler.setLevel(logging.WARNING) + warning_handler.setFormatter( + logging.Formatter(fmt="%(levelname)s: %(message)s")) + root_logger.addHandler(warning_handler) + if config.verbose_output: import re ver = pywikibot.__version__ # probably can be improved on @@ -183,6 +240,7 @@ pywikibot.output(u'Pywikipediabot r%s' % m.group(1)) pywikibot.output(u'Python %s' % sys.version) + root_logger.debug("handleArgs() completed.") return nonGlobalArgs @@ -203,11 +261,13 @@ wikipedia, wiktionary, wikitravel, ... This will override the configuration in user-config.py. --daemonize:xyz Immediately returns control to the terminal and redirects +-user:xyz Log in as user 'xyz' instead of the default username. + +-daemonize:xyz Immediately return control to the terminal and redirect stdout and stderr to xyz (only use for bots that require no input from stdin). --help Shows this help text. +-help Show this help text. -log Enable the logfile, using the default filename '%s-bot.log' Modified: branches/rewrite/pywikibot/comms/http.py =================================================================== --- branches/rewrite/pywikibot/comms/http.py 2008-12-19 12:30:48 UTC (rev 6170) +++ branches/rewrite/pywikibot/comms/http.py 2008-12-19 18:16:50 UTC (rev 6171) @@ -44,10 +44,7 @@ http_queue = Queue.Queue() cookie_jar = threadedhttp.LockableCookieJar( - config.datafilepath("%s-%s-%s.lwp" - % (config.family, - config.mylang, - config.usernames[config.family][config.mylang]))) + config.datafilepath("pywikibot.lwp")) try: cookie_jar.load() except (IOError, cookielib.LoadError): Modified: branches/rewrite/pywikibot/data/api.py =================================================================== --- branches/rewrite/pywikibot/data/api.py 2008-12-19 12:30:48 UTC (rev 6170) +++ branches/rewrite/pywikibot/data/api.py 2008-12-19 18:16:50 UTC (rev 6171) @@ -23,7 +23,7 @@ from pywikibot import login from pywikibot.exceptions import * -logger = logging.getLogger("data") +logger = logging.getLogger() lagpattern = re.compile(r"Waiting for [\d.]+: (?P<lag>\d+) seconds? lagged") @@ -198,6 +198,7 @@ logger.warning("%s, %s", uri, params) self.wait() continue + logger.debug("API response received:\n%s", rawdata) if not isinstance(rawdata, unicode): rawdata = rawdata.decode(self.site.encoding()) if rawdata.startswith(u"unknown_action"): @@ -228,11 +229,11 @@ self.site._userinfo = result['query']['userinfo'] if "warnings" in result: - modules = [k for k in result["warning"] if k != "info"] - logger.warn( - "API warning (%s): %s" - % (", ".join(modules), result['warnings']['info'])) - warnings.warn(result['warnings']['info']) + modules = [k for k in result["warnings"] if k != "info"] + for mod in modules: + logger.warning( + "API warning (%s): %s" + % (mod, result["warnings"][mod]["*"])) if "error" not in result: return result if "*" in result["error"]: @@ -558,19 +559,22 @@ login_result = login_request.submit() if u"login" not in login_result: raise RuntimeError("API login response does not have 'login' key.") - if login_result['login']['result'] != u'Success': - self._waituntil = datetime.datetime.now() + datetime.timedelta(seconds=60) - return None + if login_result['login']['result'] == u'Success': + prefix = login_result['login']['cookieprefix'] + cookies = [] + for key in ('Token', 'UserID', 'UserName'): + cookies.append("%s%s=%s" + % (prefix, key, + login_result['login']['lg'+key.lower()])) + self.username = login_result['login']['lgusername'] + return "\n".join(cookies) + elif login_result['login']['result'] == "Throttled": + self._waituntil = datetime.now() \ + + timedelta(seconds=int( + login_result["login"]["wait"]) + ) + raise APIError(code=login_result["login"]["result"], info="") - prefix = login_result['login']['cookieprefix'] - cookies = [] - for key in ('Token', 'UserID', 'UserName'): - cookies.append("%s%s=%s" - % (prefix, key, - login_result['login']['lg'+key.lower()])) - self.username = login_result['login']['lgusername'] - return "\n".join(cookies) - def storecookiedata(self, data): pywikibot.cookie_jar.save() Modified: branches/rewrite/pywikibot/login.py =================================================================== --- branches/rewrite/pywikibot/login.py 2008-12-19 12:30:48 UTC (rev 6170) +++ branches/rewrite/pywikibot/login.py 2008-12-19 18:16:50 UTC (rev 6171) @@ -50,7 +50,7 @@ from pywikibot import config from pywikibot.exceptions import * -logger = logging.getLogger("wiki") +logger = logging.getLogger() # On some wikis you are only allowed to run a bot if there is a link to @@ -191,27 +191,29 @@ logger.info(u"Logging in to %(site)s as %(name)s" % {'name': self.username, 'site': self.site}) - cookiedata = self.getCookie() - if cookiedata: - self.storecookiedata(cookiedata) - logger.info(u"Should be logged in now") - # Show a warning according to the local bot policy - if not self.botAllowed(): - logger.error( - u"Username '%(name)s' is not listed on [[%(page)s]]." - % {'name': self.username, - 'page': botList[self.site.family.name][self.site.code]}) - logger.error( -"Please make sure you are allowed to use the robot before actually using it!") - return False - return True - else: - logger.error(u"Login failed. Wrong password or CAPTCHA answer?") + try: + cookiedata = self.getCookie() + except pywikibot.data.api.APIError, e: + logger.error("Login failed (%s).", e.code) if retry: self.password = None return self.login(retry = True) else: return False + self.storecookiedata(cookiedata) + logger.info(u"Should be logged in now") +## # Show a warning according to the local bot policy +## FIXME: disabled due to recursion; need to move this to the Site object after +## login +## if not self.botAllowed(): +## logger.error( +## u"Username '%(name)s' is not listed on [[%(page)s]]." +## % {'name': self.username, +## 'page': botList[self.site.family.name][self.site.code]}) +## logger.error( +##"Please make sure you are allowed to use the robot before actually using it!") +## return False + return True def showCaptchaWindow(self, url): pass @@ -221,7 +223,7 @@ sysop = False logall = False forceLogin = False - for arg in wikipedia.handleArgs(): #FIXME + for arg in pywikibot.handleArgs(): if arg.startswith("-pass"): if len(arg) == 5: password = pywikibot.input(u'Password for all accounts:', @@ -235,7 +237,7 @@ elif arg == "-force": forceLogin = True else: - wikipedia.showHelp('login') #FIXME + pywikibot.showHelp('login') return if logall: if sysop: @@ -257,7 +259,7 @@ u' is not a valid site, please remove it from your config') else: - loginMan = LoginManager(password, sysop=sysop) + loginMan = pywikibot.data.api.LoginManager(password, sysop=sysop) loginMan.login() if __name__ == "__main__":

1 0

← Newer
1
2
3
4
5
6
7
8
...
12
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Pywikipedia-l December 2008