Bugs item #2498068, was opened at 2009-01-10 17:48
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2498068&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: interwiki
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: André Malafaya Baptista (malafaya)
Assigned to: Nobody/Anonymous (nobody)
Summary: Page editing requires full admin privileges in Windows Vista
Initial Comment:
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10)
Python 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)]
Running interwiki.py without the "Run as administrator" option in Windows Vista gives a strange behaviour:
The bot asks for bot account password, tries to login, and apparently it succeeds.
It tries to edit the page and it fails and then tries again.
Then either it succeeds in editing or I get a CAPTCHA request depending on the site it is editing.
In any case, the bot's edit is anonymous, and not using the bot account.
For the next edit, I get a bot request to login again.
Over and over again.
I experimented running it under the Vista administrator privileges (my account is an "admin" but Vista normal admin privileges are more restricted than in previous versions of Windows. You have to explicitely say you want to run in fully unsafe admin mode.
My perception is that this shouldn't affect page editing by the bot in any way, so I'm filing it as a bug, although I'm not sure it's not a Python framework problem.
Thanks.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2498068&group_…
Bugs item #2105373, was opened at 2008-09-11 10:01
Message generated for change (Comment added) made by nobody
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: replace.py crashes on redirect pages
Initial Comment:
replace.py (r5884 Python 2.5.1) crashes on redirect pages?
Getting 60 pages from wikipedia:ru...
Sleeping for 18.4 seconds, 2008-09-11 13:40:55
No changes were necessary in [[Esuvee]]
No changes were necessary in [[Et Cetera (театр)]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 708, in <module>
main()
File "D:\pywikipedia\replace.py", line 704, in main
bot.run()
File "D:\pywikipedia\replace.py", line 373, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 341, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem?
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 17:40
Message:
actually, it seems like the bot crashes not on any page containing
parentheses in the title, but on the first page with parentheses after
redirect page(s)
in this rage of pages
[http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]
it crashes after "2085 (альбом)". I guess the first "normal" page
with parenteses in the title after a number of redirects - "208
(число)" - is causing the bot's crash.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 17:27
Message:
it crashes either on redirect pages or on pages containg parentheses. what
utf-8 encoding problem might be in redirects? I guess the bot should skip
them.
if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the
following page [[2056 год]] is a redirect
([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]).
There are another three redirects, then a "normal" page with parentheses
[[205 (число)]].
replace.py crashes right after the first page:
D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055"
Getting 60 pages from wikipedia:ru...
No changes were necessary in [[2055 год]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 733, in <module> main()
File "D:\pywikipedia\replace.py", line 729, in main bot.run()
File "D:\pywikipedia\replace.py", line 383, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 351, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1:
ordinal not in range(128)
the bot continues the same way when started with "-start:205 (", so I
think it's parenthesis in the title that most likely causes the crash. can
you please double check and fix this issue?
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10)
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)]
----------------------------------------------------------------------
Comment By: André Malafaya Baptista (malafaya)
Date: 2008-09-16 13:52
Message:
I believe it has something to do with some UTF-8 encoded string not being
considered as such. But that's as far as my "knowledge" goes.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Bugs item #2105373, was opened at 2008-09-11 10:01
Message generated for change (Comment added) made by nobody
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: replace.py crashes on redirect pages
Initial Comment:
replace.py (r5884 Python 2.5.1) crashes on redirect pages?
Getting 60 pages from wikipedia:ru...
Sleeping for 18.4 seconds, 2008-09-11 13:40:55
No changes were necessary in [[Esuvee]]
No changes were necessary in [[Et Cetera (театр)]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 763, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 708, in <module>
main()
File "D:\pywikipedia\replace.py", line 704, in main
bot.run()
File "D:\pywikipedia\replace.py", line 373, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 341, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3315, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1: ordinal not in range(128)
There is a number of redirect pages following [[Et Cetera (театр)]] - http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0… - may this be a source of the problem?
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-10 17:27
Message:
it crashes either on redirect pages or on pages containg parentheses. what
utf-8 encoding problem might be in redirects? I guess the bot should skip
them.
if you look e.g. at Russian wiki [[2055 год]] is a normal page, and the
following page [[2056 год]] is a redirect
([http://ru.wikipedia.org/w/index.php?title=%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0…]).
There are another three redirects, then a "normal" page with parentheses
[[205 (число)]].
replace.py crashes right after the first page:
D:\pywikipedia>replace.py -lang:ru -fix:ru_fix -namespace:0 "-start:2055"
Getting 60 pages from wikipedia:ru...
No changes were necessary in [[2055 год]]
Traceback (most recent call last):
File "D:\pywikipedia\pagegenerators.py", line 759, in __iter__
yield loaded_page
GeneratorExit
Traceback (most recent call last):
File "D:\pywikipedia\replace.py", line 733, in <module> main()
File "D:\pywikipedia\replace.py", line 729, in main bot.run()
File "D:\pywikipedia\replace.py", line 383, in run
new_text = self.doReplacements(new_text)
File "D:\pywikipedia\replace.py", line 351, in doReplacements
allowoverlap=self.allowoverlap)
File "D:\pywikipedia\wikipedia.py", line 3413, in replaceExcept
text = text[:match.start()] + replacement + text[match.end():]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1:
ordinal not in range(128)
the bot continues the same way when started with "-start:205 (", so I
think it's parenthesis in the title that most likely causes the crash. can
you please double check and fix this issue?
Pywikipedia [http] trunk/pywikipedia (r6242, Jan 09 2009, 20:23:10)
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit
(Intel)]
----------------------------------------------------------------------
Comment By: André Malafaya Baptista (malafaya)
Date: 2008-09-16 13:52
Message:
I believe it has something to do with some UTF-8 encoded string not being
considered as such. But that's as far as my "knowledge" goes.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2105373&group_…
Feature Requests item #1722782, was opened at 2007-05-21 17:21
Message generated for change (Comment added) made by aronsson
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1722782&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Priority: 5
Private: No
Submitted By: Byrial Ole Jensen (byrial)
Assigned to: Nobody/Anonymous (nobody)
Summary: interwiki.py should follow category redirect templates
Initial Comment:
Sometimes when a category is moved, a template like [[en:template:Template:Category redirect]] (see its interwiki links for similar templates in other languages) is left at the old category page.
It would be good if interwiki.py could check for the presence of a such template and follow the redirect to the new category as given by the template argument.
----------------------------------------------------------------------
Comment By: Lars Aronsson (aronsson)
Date: 2009-01-10 02:41
Message:
Thanks, I hadn't even looked in category_redirect.py. For the moment, I
just copied the list of template names to my version of wikipedia.py so all
my changes are in one file. I have updated the list with more template
names (and more synonyms).
The detection of #REDIRECT in wikipedia.py is done in two places, using
self.site.redirectRegex() both in Page._getEditPage() and GetAll.oneDone().
These are the two places I added an "elif" branch to look for category
redirects. I don't fully understand why there needs to be two places to do
this test, but that's a matter of overall design. The naming of
redirectRegex() is also hardwired to the use of a single regex, which
doesn't scale to category redirects. Perhaps a refactoring would lead to
that function being renamed to isRedirect(). I think redirect detection
does belong in the Site object, since it depends on language-specific
synonyms to REDIRECT and to specific templates used for category redirects.
----------------------------------------------------------------------
Comment By: Russell Blau (russblau)
Date: 2009-01-09 14:17
Message:
category_redirect.py already contains a list of category redirect
templates, although only for a few sites. If it is desired to use this
capability in other bots, then the template lists should probably be moved
into the family files, and an is_category_redirect() method added to the
Category object in catlib.py, or alternatively to the Page object.
----------------------------------------------------------------------
Comment By: Lars Aronsson (aronsson)
Date: 2009-01-09 12:40
Message:
I now have some code that I believe solves this. But since I'm a beginner
in Python, I'd like someone more experienced to look at my code before it
is submitted.
----------------------------------------------------------------------
Comment By: Lars Aronsson (aronsson)
Date: 2009-01-09 01:46
Message:
The previous comment was by me. I don't know why I wasn't logged in.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-01-09 01:43
Message:
Implementing this feature involves several steps.
First the template needs to be detected. This is similar to isDisambig()
in wikipedia.py. Perhaps that function should also require isCategory(), so
the template is only detected when used in category pages. Unfortunately,
there is no equivalent to the MediaWiki:Disambiguationspage to help us find
out what the template name is in each language, so we have to list the
template translations for each language. I think that should be manageable.
I propose the new function be called isCategoryRedirect(). Then this
function needs to be introduced where isRedirect() is used. Or perhaps
isRedirect() should call it? That would save a lot of work.
Are there some situations where it would be harmful to detect this
template? Should the use of the new function be configurable?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1722782&group_…
Revision: 6241
Author: multichill
Date: 2009-01-09 20:22:10 +0000 (Fri, 09 Jan 2009)
Log Message:
-----------
Or friends at nn got a message between the categories and the iw's and they want to keep it there.
Should be a better way to fix this, but this works.
Modified Paths:
--------------
trunk/pywikipedia/add_text.py
Modified: trunk/pywikipedia/add_text.py
===================================================================
--- trunk/pywikipedia/add_text.py 2009-01-09 20:19:56 UTC (rev 6240)
+++ trunk/pywikipedia/add_text.py 2009-01-09 20:22:10 UTC (rev 6241)
@@ -67,6 +67,8 @@
'zh': u'機器人: 正在新增 %s',
}
+nn_iw_msg = u'<!--interwiki (no, sv, da first; then other languages alphabetically by name)-->'
+
class NoEnoughData(wikipedia.Error):
""" Error class for when the user doesn't specified all the data needed """
@@ -164,10 +166,16 @@
interwikiInside = wikipedia.getLanguageLinks(newtext, site)
# Removing the interwiki
newtext = wikipedia.removeLanguageLinks(newtext, site)
+ #nn got a message between the categories and the iw's and they want to keep it there, first remove it
+ if (site.language()==u'nn'):
+ newtext = newtext.replace(nn_iw_msg, '')
# Adding the text
newtext += u"\n%s" % addText
# Reputting the categories
newtext = wikipedia.replaceCategoryLinks(newtext, categoriesInside, site, True)
+ #Put the nn iw message back
+ if (site.language()==u'nn'):
+ newtext = newtext + u'\n' + nn_iw_msg
# Dealing the stars' issue
starsListInPage = list()
for star in starsList:
Revision: 6240
Author: multichill
Date: 2009-01-09 20:19:56 +0000 (Fri, 09 Jan 2009)
Log Message:
-----------
Added dotall and multiline options for the regular expressions. Info at http://docs.python.org/library/re.html#re.compile
Modified Paths:
--------------
trunk/pywikipedia/replace.py
Modified: trunk/pywikipedia/replace.py
===================================================================
--- trunk/pywikipedia/replace.py 2009-01-09 19:35:00 UTC (rev 6239)
+++ trunk/pywikipedia/replace.py 2009-01-09 20:19:56 UTC (rev 6240)
@@ -27,6 +27,11 @@
-nocase Use case insensitive regular expressions.
+-dotall Make the dot match any character at all, including a newline.
+ Without this flag, '.' will match anything except a newline.
+
+-multiline '^' and '$' will now match begin and end of each line.
+
-xmlstart (Only works with -xml) Skip all articles in the XML dump
before the one specified (may also be given as
-xmlstart:Article).
@@ -489,6 +494,10 @@
acceptall = False
# Will become True if the user inputs the commandline parameter -nocase
caseInsensitive = False
+ # Will become True if the user inputs the commandline parameter -dotall
+ dotall = False
+ # Will become True if the user inputs the commandline parameter -multiline
+ multiline = False
# Which namespaces should be processed?
# default to [] which means all namespaces will be processed
namespaces = []
@@ -555,6 +564,10 @@
recursive = True
elif arg == '-nocase':
caseInsensitive = True
+ elif arg == '-dotall':
+ dotall = True
+ elif arg == '-multiline':
+ multiline = True
elif arg.startswith('-addcat:'):
add_cat = arg[len('addcat:'):]
elif arg.startswith('-namespace:'):
@@ -641,15 +654,21 @@
exceptions = fix['exceptions']
replacements = fix['replacements']
+ #Set the regular expression flags
+ flags = re.UNICODE
+ if caseInsensitive:
+ flags = flags | re.IGNORECASE
+ if dotall:
+ flags = flags | re.DOTALL
+ if multiline:
+ flags = flags | re.MULTILINE
+
# Pre-compile all regular expressions here to save time later
for i in range(len(replacements)):
old, new = replacements[i]
if not regex:
old = re.escape(old)
- if caseInsensitive:
- oldR = re.compile(old, re.UNICODE | re.IGNORECASE)
- else:
- oldR = re.compile(old, re.UNICODE)
+ oldR = re.compile(old, flags)
replacements[i] = oldR, new
for exceptionCategory in ['title', 'require-title', 'text-contains', 'inside']:
@@ -657,12 +676,7 @@
patterns = exceptions[exceptionCategory]
if not regex:
patterns = [re.escape(pattern) for pattern in patterns]
- if caseInsensitive:
- patterns = [re.compile(pattern, re.UNICODE | re.IGNORECASE)
- for pattern in patterns]
- else:
- patterns = [re.compile(pattern, re.UNICODE)
- for pattern in patterns]
+ patterns = [re.compile(pattern, flags) for pattern in patterns]
exceptions[exceptionCategory] = patterns
if xmlFilename: