pywikibot

pywikibot@lists.wikimedia.org

6830 discussions

[Pywikipedia-l] SVN: [6253] trunk/pywikipedia/cfd.py
by cydeweys＠svn.wikimedia.org 13 Jan '09

13 Jan '09

Revision: 6253 Author: cydeweys Date: 2009-01-14 02:45:23 +0000 (Wed, 14 Jan 2009) Log Message: ----------- Fixing a minor bug where the CFDbot did not catch the NoPage exception while attempting to figure out the date link on a CFD, so it was not able to empty out redlinked categories. Modified Paths: -------------- trunk/pywikipedia/cfd.py Modified: trunk/pywikipedia/cfd.py =================================================================== --- trunk/pywikipedia/cfd.py 2009-01-13 14:05:52 UTC (rev 6252) +++ trunk/pywikipedia/cfd.py 2009-01-14 02:45:23 UTC (rev 6253) @@ -119,8 +119,12 @@ # per-day subheadings on the working page. def findDay(pageTitle, oldDay): page = wikipedia.Page(wikipedia.getSite(), "Category:" + pageTitle) - pageSrc = page.get() - m = findday.search(pageSrc) + try: + pageSrc = page.get() + m = findday.search(pageSrc) + except wikipedia.NoPage: + m = None + if (m != None): return "[[" + m.group(1) + "]]" else:

1 0

[Pywikipedia-l] [ pywikipediabot-Feature Requests-1722782 ] interwiki.py should follow category redirect templates
by SourceForge.net 13 Jan '09

13 Jan '09

Feature Requests item #1722782, was opened at 2007-05-21 11:21 Message generated for change (Settings changed) made by russblau You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1722782&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Priority: 5 Private: No Submitted By: Byrial Ole Jensen (byrial) >Assigned to: Russell Blau (russblau) Summary: interwiki.py should follow category redirect templates Initial Comment: Sometimes when a category is moved, a template like [[en:template:Template:Category redirect]] (see its interwiki links for similar templates in other languages) is left at the old category page. It would be good if interwiki.py could check for the presence of a such template and follow the redirect to the new category as given by the template argument. ---------------------------------------------------------------------- Comment By: Lars Aronsson (aronsson) Date: 2009-01-12 17:22 Message: The previous comment was a failed attempt to submit a patch. The same code is available as the attached file mydiff. Most of this patch is a list of template names, that originated in category_redirect.py and really should move to family.py or some place like that. What I've done is to add an "elif" branch in the two places, where self.site.redirectRegex() is tested, so it also looks for these templates, but only if we are in a category page. Maybe self.site shouldn't hand out a regex, but instead provide the function that tests for redirects. Feel free to refactor this. ---------------------------------------------------------------------- Comment By: Lars Aronsson (aronsson) Date: 2009-01-09 20:41 Message: Thanks, I hadn't even looked in category_redirect.py. For the moment, I just copied the list of template names to my version of wikipedia.py so all my changes are in one file. I have updated the list with more template names (and more synonyms). The detection of #REDIRECT in wikipedia.py is done in two places, using self.site.redirectRegex() both in Page._getEditPage() and GetAll.oneDone(). These are the two places I added an "elif" branch to look for category redirects. I don't fully understand why there needs to be two places to do this test, but that's a matter of overall design. The naming of redirectRegex() is also hardwired to the use of a single regex, which doesn't scale to category redirects. Perhaps a refactoring would lead to that function being renamed to isRedirect(). I think redirect detection does belong in the Site object, since it depends on language-specific synonyms to REDIRECT and to specific templates used for category redirects. ---------------------------------------------------------------------- Comment By: Russell Blau (russblau) Date: 2009-01-09 08:17 Message: category_redirect.py already contains a list of category redirect templates, although only for a few sites. If it is desired to use this capability in other bots, then the template lists should probably be moved into the family files, and an is_category_redirect() method added to the Category object in catlib.py, or alternatively to the Page object. ---------------------------------------------------------------------- Comment By: Lars Aronsson (aronsson) Date: 2009-01-09 06:40 Message: I now have some code that I believe solves this. But since I'm a beginner in Python, I'd like someone more experienced to look at my code before it is submitted. ---------------------------------------------------------------------- Comment By: Lars Aronsson (aronsson) Date: 2009-01-08 19:46 Message: The previous comment was by me. I don't know why I wasn't logged in. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-08 19:43 Message: Implementing this feature involves several steps. First the template needs to be detected. This is similar to isDisambig() in wikipedia.py. Perhaps that function should also require isCategory(), so the template is only detected when used in category pages. Unfortunately, there is no equivalent to the MediaWiki:Disambiguationspage to help us find out what the template name is in each language, so we have to list the template translations for each language. I think that should be manageable. I propose the new function be called isCategoryRedirect(). Then this function needs to be introduced where isRedirect() is used. Or perhaps isRedirect() should call it? That would save a lot of work. Are there some situations where it would be harmful to detect this template? Should the use of the new function be configurable? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1722782&group_…

1 0

[Pywikipedia-l] SVN: [6252] trunk/pywikipedia/category_redirect.py
by russblau＠svn.wikimedia.org 13 Jan '09

13 Jan '09

Revision: 6252 Author: russblau Date: 2009-01-13 14:05:52 +0000 (Tue, 13 Jan 2009) Log Message: ----------- catch exceptions; sort log output Modified Paths: -------------- trunk/pywikipedia/category_redirect.py Modified: trunk/pywikipedia/category_redirect.py =================================================================== --- trunk/pywikipedia/category_redirect.py 2009-01-13 13:38:39 UTC (rev 6251) +++ trunk/pywikipedia/category_redirect.py 2009-01-13 14:05:52 UTC (rev 6252) @@ -501,11 +501,16 @@ if match.group(1): # category redirect target starts with "Category:" - fix it text = text[ :match.start(1)] + text[match.end(1): ] - cat.put(text, - u"Robot: fixing category redirect parameter format") - self.log_text.append( - u"* Removed category prefix from parameter in %s" - % cat.aslink(textlink=True)) + try: + cat.put(text, + u"Robot: fixing category redirect parameter format") + self.log_text.append( + u"* Removed category prefix from parameter in %s" + % cat.aslink(textlink=True)) + except wikipedia.Error: + self.log_text.append( + u"* Unable to save changes to %s" + % cat.aslink(textlink=True)) # delete record entries for non-existent categories for cat_name in list(record.keys()): @@ -595,6 +600,7 @@ wikipedia.setAction(wikipedia.translate(self.site.lang, self.maint_comment)) + self.log_text.sort() self.log_page.put(u"\n==%i-%02i-%02iT%02i:%02i:%02iZ==\n" % time.gmtime()[:6] + u"\n".join(self.log_text)

1 0

[Pywikipedia-l] SVN: [6251] trunk/pywikipedia/wikipedia.py
by russblau＠svn.wikimedia.org 13 Jan '09

13 Jan '09

Revision: 6251 Author: russblau Date: 2009-01-13 13:38:39 +0000 (Tue, 13 Jan 2009) Log Message: ----------- fix interwiki problem Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2009-01-12 19:47:25 UTC (rev 6250) +++ trunk/pywikipedia/wikipedia.py 2009-01-13 13:38:39 UTC (rev 6251) @@ -578,21 +578,25 @@ points to the home wiki. If False, return an interwiki link only if needed. - If textlink is True, always return a link in text form (that - is, links to the Category: and Image: namespaces will be preceded by - a : character). (Not needed if forceInterwiki is True.) + If textlink is True, always return a link in text form (that is, + interwiki links and internal links to the Category: and Image: + namespaces will be preceded by a : character). """ if not noInterwiki and (forceInterwiki or self.site() != getSite()): + if textlink: + left = "[[:" + else: + left = "[[" if self.site().family != getSite().family \ and self.site().family.name != self.site().lang: - return u'[[:%s:%s:%s]]' % (self.site().family.name, - self.site().lang, - self.title(savetitle=True, - decode=True)) + return u'%s%s:%s:%s]]' % (left, self.site().family.name, + self.site().lang, + self.title(savetitle=True, + decode=True)) else: - return u'[[:%s:%s]]' % (self.site().lang, - self.title(savetitle=True, decode=True)) + return u'%s%s:%s]]' % (left, self.site().lang, + self.title(savetitle=True, decode=True)) elif textlink and (self.isImage() or self.isCategory()): return u'[[:%s]]' % self.title(savetitle=True, decode=True) else:

1 0

[Pywikipedia-l] [ pywikipediabot-Feature Requests-1722782 ] interwiki.py should follow category redirect templates
by SourceForge.net 12 Jan '09

12 Jan '09

1 0

[Pywikipedia-l] [ pywikipediabot-Feature Requests-1722782 ] interwiki.py should follow category redirect templates
by SourceForge.net 12 Jan '09

12 Jan '09

Feature Requests item #1722782, was opened at 2007-05-21 17:21 Message generated for change (Comment added) made by aronsson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1722782&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Priority: 5 Private: No Submitted By: Byrial Ole Jensen (byrial) Assigned to: Nobody/Anonymous (nobody) Summary: interwiki.py should follow category redirect templates Initial Comment: Sometimes when a category is moved, a template like [[en:template:Template:Category redirect]] (see its interwiki links for similar templates in other languages) is left at the old category page. It would be good if interwiki.py could check for the presence of a such template and follow the redirect to the new category as given by the template argument. ---------------------------------------------------------------------- Comment By: Lars Aronsson (aronsson) Date: 2009-01-12 23:08 Message: Index: wikipedia.py =================================================================== --- wikipedia.py (revision 6250) +++ wikipedia.py (working copy) @@ -830,6 +830,10 @@ self._redirarg = redirtarget else: raise IsRedirectPage(redirtarget) + elif self.isCategoryRedirect(): # sets _redirarg + if not get_redirect: + self._getexception = IsRedirectPage + raise IsRedirectPage, self._redirarg if self.section(): # TODO: What the hell is this? Docu please. m = re.search("\.3D\_*(\.27\.27+)?(\.5B\.5B)?\_*%s\_*(\.5B\.5B)?(\.27\.27+)?\_*\.3D" % re.escape(self.section()), sectionencode(text,self.site().encoding())) @@ -1078,6 +1082,122 @@ """Return True if this is an image description page, False otherwise.""" return self.namespace() == 6 + def isCategoryRedirect(self): + if not self.isCategory(): + return False + if not hasattr(self, "_isCategoryRedirect"): + if not hasattr(self.site(), "_categoryredirecttemplates"): + # Big table duplicated from category_redirecty.py + # Where should this list reside? In family.py? + try: + self._site._categoryredirecttemplates = { + 'wikipedia': { + 'ar': (u"تحويل تصنيف", + u"تحويلة تصنيف" + u"Category redirect", + u"تحويلة تصنيف"), + 'arz': (u'تحويل تصنيف'), + 'cs': (u'Zastaralá kategorie'), + 'da': (u'Kategoriomdirigering'), + 'de': (u'Kategorieweiterleitung'), + 'en': (u"Category redirect", + u"Category redirect3", + u"Categoryredirect", + u"Empty category", + u"CR", + u"Catredirect", + u"Cat redirect", + u"Emptycat", + u"Emptycategory", + u"Empty cat", + u"Seecat"), + 'es': (u'Categoría redirigida'), + 'eu': (u'Kategoria redirect'), + 'fa': (u'رده بهتر', + u'انتقال رده', + u'فیلم‌های امریکایی'), + 'fr': (u'Redirection de catégorie'), + 'hi': (u'श्रेणीअनुप्रेषित', + u'Categoryredirect'), + 'id': (u'Alih kategori', + u'Alihkategori'), + # 'it' has removed its template + # 'ja' is discussing to remove this template + 'ja': (u"Category redirect"), + 'ko': (u'분류 넘겨주기'), + 'mk': (u'Премести категорија'), + 'ms': (u'Pengalihan kategori', + u'Categoryredirect', + u'Category redirect'), + 'mt': (u'Redirect kategorija'), + # 'nl' has removed its template + 'no': (u"Category redirect", + u"Kategoriomdirigering", + u"Kategori-omdirigering"), + 'pl': (u'Przekierowanie kategorii', + u'Category redirect'), + 'pt': (u'Redirecionamento de categoria', + u'Redircat', + u'Redirect-categoria'), + 'ro': (u'Redirect categorie'), + 'ru': (u'Переименованная категория', + u'Categoryredirect', + u'CategoryRedirect', + u'Category redirect', + u'Catredirect'), + 'simple': (u"Category redirect", + u"Catredirect"), + 'sq': (u'Kategori e zhvendosur', + u'Category redirect'), + 'tl': (u'Category redirect'), + 'tr': (u'Kategori yönlendirme', + u'Kat redir'), + 'uk': (u'Categoryredirect'), + 'vi': (u'Đổi hướng thể loại', + u'Thể loại đổi hướng', + u'Chuyển hướng thể loại', + u'Categoryredirect', + u'Category redirect', + u'Catredirect', + u'Categoryredirect'), + 'yi': (u'קאטעגאריע אריבערפירן'), + 'zh': (u'分类重定向', + u'Cr', + u'CR', + u'Cat-redirect'), + 'zh-yue': (u'Category redirect', + u'分類彈去', + u'分類跳轉'), + }, + 'commons': { + 'commons': (u'Category redirect', + u'Categoryredirect', + u'See cat', + u'Seecat', + u'Catredirect', + u'Cat redirect', + u'CatRed', + u'Cat-red', + u'Catredir', + u'Redirect category'), + } + }[self._site.family.name][self._site.lang] + except: + self._site._categoryredirecttemplates = None + + if self._site._categoryredirecttemplates is None: + self._isCategoryRedirect = False + return False + for (t, arg) in self.templatesWithParams(): + if t in self.site()._categoryredirecttemplates: + self._isCategoryRedirect = True + # Get target + self._redirarg = self._site.namespace(self._namespace) + ":" + arg[0] + break + else: + self._isCategoryRedirect = False + return self._isCategoryRedirect + def isDisambig(self): """Return True if this is a disambiguation page, False otherwise. @@ -2958,6 +3078,8 @@ page2._revisionId = revisionId page2._editTime = timestamp section = page2.section() + page2._contents = text + m = self.site.redirectRegex().match(text) if m: ## output(u"%s is a redirect" % page2.aslink()) @@ -2966,6 +3088,9 @@ redirectto = redirectto+"#"+section page2._getexception = IsRedirectPage page2._redirarg = redirectto + elif page2.isCategoryRedirect(): + page2._getexception = IsRedirectPage + # This is used for checking deletion conflict. # Use the data loading time. page2._startTime = time.strftime('%Y%m%d%H%M%S', time.gmtime()) ---------------------------------------------------------------------- Comment By: Lars Aronsson (aronsson) Date: 2009-01-10 02:41 Message: Thanks, I hadn't even looked in category_redirect.py. For the moment, I just copied the list of template names to my version of wikipedia.py so all my changes are in one file. I have updated the list with more template names (and more synonyms). The detection of #REDIRECT in wikipedia.py is done in two places, using self.site.redirectRegex() both in Page._getEditPage() and GetAll.oneDone(). These are the two places I added an "elif" branch to look for category redirects. I don't fully understand why there needs to be two places to do this test, but that's a matter of overall design. The naming of redirectRegex() is also hardwired to the use of a single regex, which doesn't scale to category redirects. Perhaps a refactoring would lead to that function being renamed to isRedirect(). I think redirect detection does belong in the Site object, since it depends on language-specific synonyms to REDIRECT and to specific templates used for category redirects. ---------------------------------------------------------------------- Comment By: Russell Blau (russblau) Date: 2009-01-09 14:17 Message: category_redirect.py already contains a list of category redirect templates, although only for a few sites. If it is desired to use this capability in other bots, then the template lists should probably be moved into the family files, and an is_category_redirect() method added to the Category object in catlib.py, or alternatively to the Page object. ---------------------------------------------------------------------- Comment By: Lars Aronsson (aronsson) Date: 2009-01-09 12:40 Message: I now have some code that I believe solves this. But since I'm a beginner in Python, I'd like someone more experienced to look at my code before it is submitted. ---------------------------------------------------------------------- Comment By: Lars Aronsson (aronsson) Date: 2009-01-09 01:46 Message: The previous comment was by me. I don't know why I wasn't logged in. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-09 01:43 Message: Implementing this feature involves several steps. First the template needs to be detected. This is similar to isDisambig() in wikipedia.py. Perhaps that function should also require isCategory(), so the template is only detected when used in category pages. Unfortunately, there is no equivalent to the MediaWiki:Disambiguationspage to help us find out what the template name is in each language, so we have to list the template translations for each language. I think that should be manageable. I propose the new function be called isCategoryRedirect(). Then this function needs to be introduced where isRedirect() is used. Or perhaps isRedirect() should call it? That would save a lot of work. Are there some situations where it would be harmful to detect this template? Should the use of the new function be configurable? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1722782&group_…

1 0

[Pywikipedia-l] [ pywikipediabot-Bugs-2105373 ] replace.py crashes
by SourceForge.net 12 Jan '09

12 Jan '09

Bugs item #2105373, was opened at 2008-09-11 06:01 Message generated for change (Settings changed) made by russblau You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: General Group: None >Status: Closed >Resolution: Invalid Priority: 5 Private: No Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: replace.py crashes Initial Comment: replace.py (r5884 Python 2.5.1) crashes on redirect pages? Getting 60 pages from wikipedia:ru... Sleeping for 18.4 seconds, 2008-09-11 13:40:55 No changes were necessary in [[Esuvee]] No changes were necessary in [[Et Cetera (театр)]] Traceback (most recent call last): File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__ yield loaded_page GeneratorExit Traceback (most recent call last): File "D:\pywikipedia\replace.py", line 708, in <module> main() File "D:\pywikipedia\replace.py", line 704, in main bot.run() File "D:\pywikipedia\replace.py", line 373, in run new_text = self.doReplacements(new_text) File "D:\pywikipedia\replace.py", line 341, in doReplacements allowoverlap=self.allowoverlap) File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept text = text[:match.start()] + replacement + text[match.end():] UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128) There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem? ---------------------------------------------------------------------- >Comment By: Russell Blau (russblau) Date: 2009-01-12 15:01 Message: Also, note that in your second replacement, the second string is not marked as Unicode (by using u before the '), which might be causing an encoding/decoding problem: (ur'\bн\. ?э\.?', ur'н.\u00A0э.'), # н. э. (ur'(Д|д)(о|\.) н\. ?э\.?', '\1о\u00A0н.\u00A0э.'), # до н. э. Anyway, I'm closing this for now since we haven't identified a bug in the replace.py script (although I'm not 100% convinced that there isn't a problem here). ---------------------------------------------------------------------- Comment By: NicDumZ — Nicolas Dumazet (nicdumz) Date: 2009-01-12 13:08 Message: Well, depending on how you created your user-fixes.py, the file might be misencoded. ur' н\. ?э\.\|?.*\]\]' however can match pretty much everything. Since you put a ".*", the exception will match 'н. э. blablabla [[link]]' for example. Are you sure you wanted to do this? Given the comment "don't change to non-breaking space within links" I think you might want the line to be something along the lines of ur'\[\[[^\]]* н\. ?э\.[^\]]*\]\]' ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-12 12:02 Message: Hi guys! I found the string that caused the bot to crash. It was this line: ur' н\. ?э\.\|?.*\]\]', # don't change to non-breaking space within links in ### EXCEPTIONS ### section of my user-fixes.py I have disabled it, and the bot works just fine. But can you please tell me what's wrong in this string? It was supposed to restrict these two replacements (ur'\bн\. ?э\.?', ur'н.\u00A0э.'), # н. э. (ur'(Д|д)(о|\.) н\. ?э\.?', '\1о\u00A0н.\u00A0э.'), # до н. э. Thanks! ---------------------------------------------------------------------- Comment By: NicDumZ — Nicolas Dumazet (nicdumz) Date: 2009-01-11 11:06 Message: I fully support russ' comment. If the bug occurs again, please also post the content of your fixes.py, and particularly your ru_fix section... ---------------------------------------------------------------------- Comment By: Russell Blau (russblau) Date: 2009-01-10 18:18 Message: Please try upgrading to SVN version (r6248) and see if the problem occurs again. If so, please post the command line you were running when the script crashed. The Python message indicates that this is a Unicode error, so I do not think it has anything to do with whether a page is a redirect; it probably has to do with some of the text in the page body. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 13:46 Message: I have further investigated the issue. It seems like the bot always crashes on pages containing "(число)" in the title (that is "number" in Russian), e.g. http://ru.wikipedia.org/wiki/221_(число). May someone figure out what's wrong in this byte sequence? Or the problem is not in the title but within the page body? I see nothing extraordinary there. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 12:40 Message: actually, it seems like the bot crashes not on any page containing parentheses in the title, but on the first page with parentheses after redirect page(s) in this rage of pages [http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…] it crashes after "2085 (альбом)". I guess the first "normal" page with parenteses in the title after a number of redirects - "208 (число)" - is causing the bot's crash. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 12:27 Message: it crashes either on redirect pages or on pages containg parentheses. what utf-8 encoding problem might be in redirects? I guess the bot should skip them. if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the following page [[2056 год]] is a redirect ([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]). There are another three redirects, then a "normal" page with parentheses [[205 (число)]]. replace.py crashes right after the first page: D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055" Getting 60 pages from wikipedia:ru... No changes were necessary in [[2055 год]] Traceback (most recent call last): File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__ yield loaded_page GeneratorExit Traceback (most recent call last): File "D:\pywikipedia\replace.py", line 733, in <module> main() File "D:\pywikipedia\replace.py", line 729, in main bot.run() File "D:\pywikipedia\replace.py", line 383, in run new_text = self.doReplacements(new_text) File "D:\pywikipedia\replace.py", line 351, in doReplacements allowoverlap=self.allowoverlap) File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept text = text[:match.start()] + replacement + text[match.end():] UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128) the bot continues the same way when started with "-start:205 (", so I think it's parenthesis in the title that most likely causes the crash. can you please double check and fix this issue? Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10) Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] ---------------------------------------------------------------------- Comment By: André Malafaya Baptista (malafaya) Date: 2008-09-16 09:52 Message: I believe it has something to do with some UTF-8 encoded string not being considered as such. But that's as far as my "knowledge" goes. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…

1 0

[Pywikipedia-l] SVN: [6249] trunk/pywikipedia/wikipedia.py
by russblau＠svn.wikimedia.org 12 Jan '09

12 Jan '09

Revision: 6249 Author: russblau Date: 2009-01-12 13:44:28 +0000 (Mon, 12 Jan 2009) Log Message: ----------- handleArgs() bug and aslink() formatting Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2009-01-10 23:14:13 UTC (rev 6248) +++ trunk/pywikipedia/wikipedia.py 2009-01-12 13:44:28 UTC (rev 6249) @@ -586,12 +586,13 @@ if not noInterwiki and (forceInterwiki or self.site() != getSite()): if self.site().family != getSite().family \ and self.site().family.name != self.site().lang: - return u'[[%s:%s:%s]]' % (self.site().family.name, - self.site().lang, - self.title(savetitle=True,decode=True)) + return u'[[:%s:%s:%s]]' % (self.site().family.name, + self.site().lang, + self.title(savetitle=True, + decode=True)) else: - return u'[[%s:%s]]' % (self.site().lang, - self.title(savetitle=True,decode=True)) + return u'[[:%s:%s]]' % (self.site().lang, + self.title(savetitle=True, decode=True)) elif textlink and (self.isImage() or self.isCategory()): return u'[[:%s]]' % self.title(savetitle=True, decode=True) else: @@ -6194,14 +6195,14 @@ global default_code, default_family, verbose # get commandline arguments if not args: - args = sys.argv + args = sys.argv[1:] # get the name of the module calling this function. This is # required because the -help option loads the module's docstring and because # the module name will be used for the filename of the log. # TODO: check if the following line is platform-independent moduleName = calledModuleName() nonGlobalArgs = [] - for arg in args[1:]: + for arg in args: arg = decodeArg(arg) if arg == '-help': showHelp(moduleName)

2 1

[Pywikipedia-l] SVN: [6250] trunk/pywikipedia/wikipedia.py
by valhallasw＠svn.wikimedia.org 12 Jan '09

12 Jan '09

Revision: 6250 Author: valhallasw Date: 2009-01-12 19:47:25 +0000 (Mon, 12 Jan 2009) Log Message: ----------- Interwiki bugfix for bug introduced in r6249 Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2009-01-12 13:44:28 UTC (rev 6249) +++ trunk/pywikipedia/wikipedia.py 2009-01-12 19:47:25 UTC (rev 6250) @@ -1,4 +1,4 @@ -# -*- coding: utf-8 -*- +# -*- coding: utf-8 -*- """ Library to get and put pages on a MediaWiki. @@ -3643,7 +3643,7 @@ s = [] for site in ar: try: - link = links[site].aslink(forceInterwiki=True) + link = links[site].aslink(forceInterwiki=True).replace('[[:', '[[') s.append(link) except AttributeError: s.append(getSite(site).linkto(links[site], othersite=insite))

1 0

[Pywikipedia-l] [ pywikipediabot-Bugs-2105373 ] replace.py crashes
by SourceForge.net 12 Jan '09

12 Jan '09

Bugs item #2105373, was opened at 2008-09-11 12:01 Message generated for change (Comment added) made by nicdumz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: General Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: replace.py crashes Initial Comment: replace.py (r5884 Python 2.5.1) crashes on redirect pages? Getting 60 pages from wikipedia:ru... Sleeping for 18.4 seconds, 2008-09-11 13:40:55 No changes were necessary in [[Esuvee]] No changes were necessary in [[Et Cetera (театр)]] Traceback (most recent call last): File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__ yield loaded_page GeneratorExit Traceback (most recent call last): File "D:\pywikipedia\replace.py", line 708, in <module> main() File "D:\pywikipedia\replace.py", line 704, in main bot.run() File "D:\pywikipedia\replace.py", line 373, in run new_text = self.doReplacements(new_text) File "D:\pywikipedia\replace.py", line 341, in doReplacements allowoverlap=self.allowoverlap) File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept text = text[:match.start()] + replacement + text[match.end():] UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128) There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem? ---------------------------------------------------------------------- >Comment By: NicDumZ — Nicolas Dumazet (nicdumz) Date: 2009-01-12 19:08 Message: Well, depending on how you created your user-fixes.py, the file might be misencoded. ur' н\. ?э\.\|?.*\]\]' however can match pretty much everything. Since you put a ".*", the exception will match 'н. э. blablabla [[link]]' for example. Are you sure you wanted to do this? Given the comment "don't change to non-breaking space within links" I think you might want the line to be something along the lines of ur'\[\[[^\]]* н\. ?э\.[^\]]*\]\]' ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-12 18:02 Message: Hi guys! I found the string that caused the bot to crash. It was this line: ur' н\. ?э\.\|?.*\]\]', # don't change to non-breaking space within links in ### EXCEPTIONS ### section of my user-fixes.py I have disabled it, and the bot works just fine. But can you please tell me what's wrong in this string? It was supposed to restrict these two replacements (ur'\bн\. ?э\.?', ur'н.\u00A0э.'), # н. э. (ur'(Д|д)(о|\.) н\. ?э\.?', '\1о\u00A0н.\u00A0э.'), # до н. э. Thanks! ---------------------------------------------------------------------- Comment By: NicDumZ — Nicolas Dumazet (nicdumz) Date: 2009-01-11 17:06 Message: I fully support russ' comment. If the bug occurs again, please also post the content of your fixes.py, and particularly your ru_fix section... ---------------------------------------------------------------------- Comment By: Russell Blau (russblau) Date: 2009-01-11 00:18 Message: Please try upgrading to SVN version (r6248) and see if the problem occurs again. If so, please post the command line you were running when the script crashed. The Python message indicates that this is a Unicode error, so I do not think it has anything to do with whether a page is a redirect; it probably has to do with some of the text in the page body. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 19:46 Message: I have further investigated the issue. It seems like the bot always crashes on pages containing "(число)" in the title (that is "number" in Russian), e.g. http://ru.wikipedia.org/wiki/221_(число). May someone figure out what's wrong in this byte sequence? Or the problem is not in the title but within the page body? I see nothing extraordinary there. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 18:40 Message: actually, it seems like the bot crashes not on any page containing parentheses in the title, but on the first page with parentheses after redirect page(s) in this rage of pages [http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…] it crashes after "2085 (альбом)". I guess the first "normal" page with parenteses in the title after a number of redirects - "208 (число)" - is causing the bot's crash. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 18:27 Message: it crashes either on redirect pages or on pages containg parentheses. what utf-8 encoding problem might be in redirects? I guess the bot should skip them. if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the following page [[2056 год]] is a redirect ([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]). There are another three redirects, then a "normal" page with parentheses [[205 (число)]]. replace.py crashes right after the first page: D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055" Getting 60 pages from wikipedia:ru... No changes were necessary in [[2055 год]] Traceback (most recent call last): File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__ yield loaded_page GeneratorExit Traceback (most recent call last): File "D:\pywikipedia\replace.py", line 733, in <module> main() File "D:\pywikipedia\replace.py", line 729, in main bot.run() File "D:\pywikipedia\replace.py", line 383, in run new_text = self.doReplacements(new_text) File "D:\pywikipedia\replace.py", line 351, in doReplacements allowoverlap=self.allowoverlap) File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept text = text[:match.start()] + replacement + text[match.end():] UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128) the bot continues the same way when started with "-start:205 (", so I think it's parenthesis in the title that most likely causes the crash. can you please double check and fix this issue? Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10) Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] ---------------------------------------------------------------------- Comment By: André Malafaya Baptista (malafaya) Date: 2008-09-16 15:52 Message: I believe it has something to do with some UTF-8 encoded string not being considered as such. But that's as far as my "knowledge" goes. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot