Bugs item #2105373, was opened at 2008-09-11 12:01
Message generated for change (Comment added) made by nicdumz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: General
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: replace.py crashes
Initial Comment:
replace.py (r5884 Python 2.5.1) crashes on redirect pages?
Getting 60 pages from wikipedia:ru...
Sleeping for 18.4 seconds, 2008-09-11 13:40:55
No changes were necessary in [[Esuvee]]
No changes were necessary in [[Et Cetera (театр)]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 708, in <module>
main()
File "D:\pywikipedia\replace.py", line 704, in main
bot.run()
File "D:\pywikipedia\replace.py", line 373, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 341, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem?
----------------------------------------------------------------------
>Comment By: NicDumZ — Nicolas Dumazet (nicdumz)
Date: 2009-01-12 19:08
Message:
Well, depending on how you created your user-fixes.py, the file might be
misencoded.
ur' н\. ?э\.\|?.*\]\]' however can match pretty much everything. Since
you put a ".*", the exception will match 'н. э. blablabla [[link]]' for
example. Are you sure you wanted to do this?
Given the comment "don't change to non-breaking space within
links" I think you might want the line to be something along the lines of
ur'\[\[[^\]]* н\. ?э\.[^\]]*\]\]'
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-12 18:02
Message:
Hi guys! I found the string that caused the bot to crash. It was this
line:
ur' н\. ?э\.\|?.*\]\]', # don't change to non-breaking space within
links
in ### EXCEPTIONS ### section of my user-fixes.py
I have disabled it, and the bot works just fine.
But can you please tell me what's wrong in this string?
It was supposed to restrict these two replacements
(ur'\bн\. ?э\.?', ur'н.\u00A0э.'), # н. э.
(ur'(Д|д)(о|\.) н\. ?э\.?', '\1о\u00A0н.\u00A0э.'), #
до н. э.
Thanks!
----------------------------------------------------------------------
Comment By: NicDumZ — Nicolas Dumazet (nicdumz)
Date: 2009-01-11 17:06
Message:
I fully support russ' comment. If the bug occurs again, please also post
the content of your fixes.py, and particularly your ru_fix section...
----------------------------------------------------------------------
Comment By: Russell Blau (russblau)
Date: 2009-01-11 00:18
Message:
Please try upgrading to SVN version (r6248) and see if the problem occurs
again. If so, please post the command line you were running when the
script crashed.
The Python message indicates that this is a Unicode error, so I do not
think it has anything to do with whether a page is a redirect; it probably
has to do with some of the text in the page body.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 19:46
Message:
I have further investigated the issue. It seems like the bot always crashes
on pages containing "(число)" in the title (that is "number" in
Russian), e.g. http://ru.wikipedia.org/wiki/221_(число). May someone
figure out what's wrong in this byte sequence? Or the problem is not in the
title but within the page body? I see nothing extraordinary there.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 18:40
Message:
actually, it seems like the bot crashes not on any page containing
parentheses in the title, but on the first page with parentheses after
redirect page(s)
in this rage of pages
[http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]
it crashes after "2085 (альбом)". I guess the first "normal" page
with parenteses in the title after a number of redirects - "208
(число)" - is causing the bot's crash.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 18:27
Message:
it crashes either on redirect pages or on pages containg parentheses. what
utf-8 encoding problem might be in redirects? I guess the bot should skip
them.
if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the
following page [[2056 год]] is a redirect
([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]).
There are another three redirects, then a "normal" page with parentheses
[[205 (число)]].
replace.py crashes right after the first page:
D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055"
Getting 60 pages from wikipedia:ru...
No changes were necessary in [[2055 год]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 733, in <module> main()
File "D:\pywikipedia\replace.py", line 729, in main bot.run()
File "D:\pywikipedia\replace.py", line 383, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 351, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1:
ordinal not in range(128)
the bot continues the same way when started with "-start:205 (", so I
think it's parenthesis in the title that most likely causes the crash. can
you please double check and fix this issue?
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10)
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)]
----------------------------------------------------------------------
Comment By: André Malafaya Baptista (malafaya)
Date: 2008-09-16 15:52
Message:
I believe it has something to do with some UTF-8 encoded string not being
considered as such. But that's as far as my "knowledge" goes.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Bugs item #2105373, was opened at 2008-09-11 10:01
Message generated for change (Comment added) made by nobody
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: General
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: replace.py crashes
Initial Comment:
replace.py (r5884 Python 2.5.1) crashes on redirect pages?
Getting 60 pages from wikipedia:ru...
Sleeping for 18.4 seconds, 2008-09-11 13:40:55
No changes were necessary in [[Esuvee]]
No changes were necessary in [[Et Cetera (театр)]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 708, in <module>
main()
File "D:\pywikipedia\replace.py", line 704, in main
bot.run()
File "D:\pywikipedia\replace.py", line 373, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 341, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem?
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-12 17:02
Message:
Hi guys! I found the string that caused the bot to crash. It was this
line:
ur' н\. ?э\.\|?.*\]\]', # don't change to non-breaking space within
links
in ### EXCEPTIONS ### section of my user-fixes.py
I have disabled it, and the bot works just fine.
But can you please tell me what's wrong in this string?
It was supposed to restrict these two replacements
(ur'\bн\. ?э\.?', ur'н.\u00A0э.'), # н. э.
(ur'(Д|д)(о|\.) н\. ?э\.?', '\1о\u00A0н.\u00A0э.'), #
до н. э.
Thanks!
----------------------------------------------------------------------
Comment By: NicDumZ — Nicolas Dumazet (nicdumz)
Date: 2009-01-11 16:06
Message:
I fully support russ' comment. If the bug occurs again, please also post
the content of your fixes.py, and particularly your ru_fix section...
----------------------------------------------------------------------
Comment By: Russell Blau (russblau)
Date: 2009-01-10 23:18
Message:
Please try upgrading to SVN version (r6248) and see if the problem occurs
again. If so, please post the command line you were running when the
script crashed.
The Python message indicates that this is a Unicode error, so I do not
think it has anything to do with whether a page is a redirect; it probably
has to do with some of the text in the page body.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 18:46
Message:
I have further investigated the issue. It seems like the bot always crashes
on pages containing "(число)" in the title (that is "number" in
Russian), e.g. http://ru.wikipedia.org/wiki/221_(число). May someone
figure out what's wrong in this byte sequence? Or the problem is not in the
title but within the page body? I see nothing extraordinary there.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 17:40
Message:
actually, it seems like the bot crashes not on any page containing
parentheses in the title, but on the first page with parentheses after
redirect page(s)
in this rage of pages
[http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]
it crashes after "2085 (альбом)". I guess the first "normal" page
with parenteses in the title after a number of redirects - "208
(число)" - is causing the bot's crash.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 17:27
Message:
it crashes either on redirect pages or on pages containg parentheses. what
utf-8 encoding problem might be in redirects? I guess the bot should skip
them.
if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the
following page [[2056 год]] is a redirect
([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]).
There are another three redirects, then a "normal" page with parentheses
[[205 (число)]].
replace.py crashes right after the first page:
D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055"
Getting 60 pages from wikipedia:ru...
No changes were necessary in [[2055 год]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 733, in <module> main()
File "D:\pywikipedia\replace.py", line 729, in main bot.run()
File "D:\pywikipedia\replace.py", line 383, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 351, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1:
ordinal not in range(128)
the bot continues the same way when started with "-start:205 (", so I
think it's parenthesis in the title that most likely causes the crash. can
you please double check and fix this issue?
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10)
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)]
----------------------------------------------------------------------
Comment By: André Malafaya Baptista (malafaya)
Date: 2008-09-16 13:52
Message:
I believe it has something to do with some UTF-8 encoded string not being
considered as such. But that's as far as my "knowledge" goes.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Bugs item #2105373, was opened at 2008-09-11 12:01
Message generated for change (Comment added) made by nicdumz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: General
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: replace.py crashes
Initial Comment:
replace.py (r5884 Python 2.5.1) crashes on redirect pages?
Getting 60 pages from wikipedia:ru...
Sleeping for 18.4 seconds, 2008-09-11 13:40:55
No changes were necessary in [[Esuvee]]
No changes were necessary in [[Et Cetera (театр)]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 708, in <module>
main()
File "D:\pywikipedia\replace.py", line 704, in main
bot.run()
File "D:\pywikipedia\replace.py", line 373, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 341, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem?
----------------------------------------------------------------------
>Comment By: NicDumZ — Nicolas Dumazet (nicdumz)
Date: 2009-01-11 17:06
Message:
I fully support russ' comment. If the bug occurs again, please also post
the content of your fixes.py, and particularly your ru_fix section...
----------------------------------------------------------------------
Comment By: Russell Blau (russblau)
Date: 2009-01-11 00:18
Message:
Please try upgrading to SVN version (r6248) and see if the problem occurs
again. If so, please post the command line you were running when the
script crashed.
The Python message indicates that this is a Unicode error, so I do not
think it has anything to do with whether a page is a redirect; it probably
has to do with some of the text in the page body.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 19:46
Message:
I have further investigated the issue. It seems like the bot always crashes
on pages containing "(число)" in the title (that is "number" in
Russian), e.g. http://ru.wikipedia.org/wiki/221_(число). May someone
figure out what's wrong in this byte sequence? Or the problem is not in the
title but within the page body? I see nothing extraordinary there.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 18:40
Message:
actually, it seems like the bot crashes not on any page containing
parentheses in the title, but on the first page with parentheses after
redirect page(s)
in this rage of pages
[http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]
it crashes after "2085 (альбом)". I guess the first "normal" page
with parenteses in the title after a number of redirects - "208
(число)" - is causing the bot's crash.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 18:27
Message:
it crashes either on redirect pages or on pages containg parentheses. what
utf-8 encoding problem might be in redirects? I guess the bot should skip
them.
if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the
following page [[2056 год]] is a redirect
([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]).
There are another three redirects, then a "normal" page with parentheses
[[205 (число)]].
replace.py crashes right after the first page:
D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055"
Getting 60 pages from wikipedia:ru...
No changes were necessary in [[2055 год]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 733, in <module> main()
File "D:\pywikipedia\replace.py", line 729, in main bot.run()
File "D:\pywikipedia\replace.py", line 383, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 351, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1:
ordinal not in range(128)
the bot continues the same way when started with "-start:205 (", so I
think it's parenthesis in the title that most likely causes the crash. can
you please double check and fix this issue?
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10)
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)]
----------------------------------------------------------------------
Comment By: André Malafaya Baptista (malafaya)
Date: 2008-09-16 15:52
Message:
I believe it has something to do with some UTF-8 encoded string not being
considered as such. But that's as far as my "knowledge" goes.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Bugs item #2105373, was opened at 2008-09-11 06:01
Message generated for change (Comment added) made by russblau
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
>Category: General
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
>Summary: replace.py crashes
Initial Comment:
replace.py (r5884 Python 2.5.1) crashes on redirect pages?
Getting 60 pages from wikipedia:ru...
Sleeping for 18.4 seconds, 2008-09-11 13:40:55
No changes were necessary in [[Esuvee]]
No changes were necessary in [[Et Cetera (театр)]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 708, in <module>
main()
File "D:\pywikipedia\replace.py", line 704, in main
bot.run()
File "D:\pywikipedia\replace.py", line 373, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 341, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem?
----------------------------------------------------------------------
>Comment By: Russell Blau (russblau)
Date: 2009-01-10 18:18
Message:
Please try upgrading to SVN version (r6248) and see if the problem occurs
again. If so, please post the command line you were running when the
script crashed.
The Python message indicates that this is a Unicode error, so I do not
think it has anything to do with whether a page is a redirect; it probably
has to do with some of the text in the page body.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 13:46
Message:
I have further investigated the issue. It seems like the bot always crashes
on pages containing "(число)" in the title (that is "number" in
Russian), e.g. http://ru.wikipedia.org/wiki/221_(число). May someone
figure out what's wrong in this byte sequence? Or the problem is not in the
title but within the page body? I see nothing extraordinary there.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 12:40
Message:
actually, it seems like the bot crashes not on any page containing
parentheses in the title, but on the first page with parentheses after
redirect page(s)
in this rage of pages
[http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]
it crashes after "2085 (альбом)". I guess the first "normal" page
with parenteses in the title after a number of redirects - "208
(число)" - is causing the bot's crash.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 12:27
Message:
it crashes either on redirect pages or on pages containg parentheses. what
utf-8 encoding problem might be in redirects? I guess the bot should skip
them.
if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the
following page [[2056 год]] is a redirect
([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]).
There are another three redirects, then a "normal" page with parentheses
[[205 (число)]].
replace.py crashes right after the first page:
D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055"
Getting 60 pages from wikipedia:ru...
No changes were necessary in [[2055 год]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 733, in <module> main()
File "D:\pywikipedia\replace.py", line 729, in main bot.run()
File "D:\pywikipedia\replace.py", line 383, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 351, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1:
ordinal not in range(128)
the bot continues the same way when started with "-start:205 (", so I
think it's parenthesis in the title that most likely causes the crash. can
you please double check and fix this issue?
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10)
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)]
----------------------------------------------------------------------
Comment By: André Malafaya Baptista (malafaya)
Date: 2008-09-16 09:52
Message:
I believe it has something to do with some UTF-8 encoded string not being
considered as such. But that's as far as my "knowledge" goes.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Bugs item #2105373, was opened at 2008-09-11 10:01
Message generated for change (Comment added) made by nobody
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: replace.py crashes on redirect pages
Initial Comment:
replace.py (r5884 Python 2.5.1) crashes on redirect pages?
Getting 60 pages from wikipedia:ru...
Sleeping for 18.4 seconds, 2008-09-11 13:40:55
No changes were necessary in [[Esuvee]]
No changes were necessary in [[Et Cetera (театр)]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 708, in <module>
main()
File "D:\pywikipedia\replace.py", line 704, in main
bot.run()
File "D:\pywikipedia\replace.py", line 373, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 341, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem?
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 18:46
Message:
I have further investigated the issue. It seems like the bot always crashes
on pages containing "(число)" in the title (that is "number" in
Russian), e.g. http://ru.wikipedia.org/wiki/221_(число). May someone
figure out what's wrong in this byte sequence? Or the problem is not in the
title but within the page body? I see nothing extraordinary there.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 17:40
Message:
actually, it seems like the bot crashes not on any page containing
parentheses in the title, but on the first page with parentheses after
redirect page(s)
in this rage of pages
[http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]
it crashes after "2085 (альбом)". I guess the first "normal" page
with parenteses in the title after a number of redirects - "208
(число)" - is causing the bot's crash.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 17:27
Message:
it crashes either on redirect pages or on pages containg parentheses. what
utf-8 encoding problem might be in redirects? I guess the bot should skip
them.
if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the
following page [[2056 год]] is a redirect
([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]).
There are another three redirects, then a "normal" page with parentheses
[[205 (число)]].
replace.py crashes right after the first page:
D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055"
Getting 60 pages from wikipedia:ru...
No changes were necessary in [[2055 год]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 733, in <module> main()
File "D:\pywikipedia\replace.py", line 729, in main bot.run()
File "D:\pywikipedia\replace.py", line 383, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 351, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1:
ordinal not in range(128)
the bot continues the same way when started with "-start:205 (", so I
think it's parenthesis in the title that most likely causes the crash. can
you please double check and fix this issue?
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10)
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)]
----------------------------------------------------------------------
Comment By: André Malafaya Baptista (malafaya)
Date: 2008-09-16 13:52
Message:
I believe it has something to do with some UTF-8 encoded string not being
considered as such. But that's as far as my "knowledge" goes.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…