Bugs item #2105373, was opened at 2008-09-11 06:01 Message generated for change (Settings changed) made by russblau You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373...
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: General Group: None
Status: Closed Resolution: Invalid
Priority: 5 Private: No Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: replace.py crashes
Initial Comment: replace.py (r5884 Python 2.5.1) crashes on redirect pages?
Getting 60 pages from wikipedia:ru... Sleeping for 18.4 seconds, 2008-09-11 13:40:55 No changes were necessary in [[Esuvee]] No changes were necessary in [[Et Cetera (театр)]] Traceback (most recent call last): File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__ yield loaded_page GeneratorExit
Traceback (most recent call last): File "D:\pywikipedia\replace.py", line 708, in <module> main() File "D:\pywikipedia\replace.py", line 704, in main bot.run() File "D:\pywikipedia\replace.py", line 373, in run new_text = self.doReplacements(new_text) File "D:\pywikipedia\replace.py", line 341, in doReplacements allowoverlap=self.allowoverlap) File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept text = text[:match.start()] + replacement + text[match.end():] UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%... - may this be a source of the problem?
----------------------------------------------------------------------
Comment By: Russell Blau (russblau)
Date: 2009-01-12 15:01
Message: Also, note that in your second replacement, the second string is not marked as Unicode (by using u before the '), which might be causing an encoding/decoding problem:
(ur'\bн. ?э.?', ur'н.\u00A0э.'), # н. э. (ur'(Д|д)(о|.) н. ?э.?', '\1о\u00A0н.\u00A0э.'), # до н. э.
Anyway, I'm closing this for now since we haven't identified a bug in the replace.py script (although I'm not 100% convinced that there isn't a problem here).
----------------------------------------------------------------------
Comment By: NicDumZ — Nicolas Dumazet (nicdumz) Date: 2009-01-12 13:08
Message: Well, depending on how you created your user-fixes.py, the file might be misencoded.
ur' н. ?э.|?.*]]' however can match pretty much everything. Since you put a ".*", the exception will match 'н. э. blablabla [[link]]' for example. Are you sure you wanted to do this? Given the comment "don't change to non-breaking space within links" I think you might want the line to be something along the lines of ur'[[[^]]* н. ?э.[^]]*]]'
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody) Date: 2009-01-12 12:02
Message: Hi guys! I found the string that caused the bot to crash. It was this line:
ur' н. ?э.|?.*]]', # don't change to non-breaking space within links
in ### EXCEPTIONS ### section of my user-fixes.py
I have disabled it, and the bot works just fine.
But can you please tell me what's wrong in this string?
It was supposed to restrict these two replacements
(ur'\bн. ?э.?', ur'н.\u00A0э.'), # н. э. (ur'(Д|д)(о|.) н. ?э.?', '\1о\u00A0н.\u00A0э.'), # до н. э.
Thanks!
----------------------------------------------------------------------
Comment By: NicDumZ — Nicolas Dumazet (nicdumz) Date: 2009-01-11 11:06
Message: I fully support russ' comment. If the bug occurs again, please also post the content of your fixes.py, and particularly your ru_fix section...
----------------------------------------------------------------------
Comment By: Russell Blau (russblau) Date: 2009-01-10 18:18
Message: Please try upgrading to SVN version (r6248) and see if the problem occurs again. If so, please post the command line you were running when the script crashed.
The Python message indicates that this is a Unicode error, so I do not think it has anything to do with whether a page is a redirect; it probably has to do with some of the text in the page body.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 13:46
Message: I have further investigated the issue. It seems like the bot always crashes on pages containing "(число)" in the title (that is "number" in Russian), e.g. http://ru.wikipedia.org/wiki/221_(%D1%87%D0%B8%D1%81%D0%BB%D0%BE). May someone figure out what's wrong in this byte sequence? Or the problem is not in the title but within the page body? I see nothing extraordinary there.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 12:40
Message: actually, it seems like the bot crashes not on any page containing parentheses in the title, but on the first page with parentheses after redirect page(s)
in this rage of pages [http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%...] it crashes after "2085 (альбом)". I guess the first "normal" page with parenteses in the title after a number of redirects - "208 (число)" - is causing the bot's crash.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody) Date: 2009-01-10 12:27
Message: it crashes either on redirect pages or on pages containg parentheses. what utf-8 encoding problem might be in redirects? I guess the bot should skip them.
if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the following page [[2056 год]] is a redirect ([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%...]). There are another three redirects, then a "normal" page with parentheses [[205 (число)]].
replace.py crashes right after the first page:
D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055" Getting 60 pages from wikipedia:ru... No changes were necessary in [[2055 год]] Traceback (most recent call last): File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__ yield loaded_page GeneratorExit
Traceback (most recent call last): File "D:\pywikipedia\replace.py", line 733, in <module> main() File "D:\pywikipedia\replace.py", line 729, in main bot.run() File "D:\pywikipedia\replace.py", line 383, in run new_text = self.doReplacements(new_text) File "D:\pywikipedia\replace.py", line 351, in doReplacements allowoverlap=self.allowoverlap) File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept text = text[:match.start()] + replacement + text[match.end():] UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
the bot continues the same way when started with "-start:205 (", so I think it's parenthesis in the title that most likely causes the crash. can you please double check and fix this issue?
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10) Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)]
----------------------------------------------------------------------
Comment By: André Malafaya Baptista (malafaya) Date: 2008-09-16 09:52
Message: I believe it has something to do with some UTF-8 encoded string not being considered as such. But that's as far as my "knowledge" goes.
----------------------------------------------------------------------
You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373...
pywikipedia-l@lists.wikimedia.org