http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9923
Revision: 9923
Author: binbot
Date: 2012-02-22 18:12:11 +0000 (Wed, 22 Feb 2012)
Log Message:
-----------
Follow-up to r9922: the exceptions function given with 'include' key will take the name of the fix as argument.
Modified Paths:
--------------
trunk/pywikipedia/replace.py
Modified: trunk/pywikipedia/replace.py
===================================================================
--- trunk/pywikipedia/replace.py 2012-02-22 17:26:32 UTC (rev 9922)
+++ trunk/pywikipedia/replace.py 2012-02-22 18:12:11 UTC (rev 9923)
@@ -317,12 +317,13 @@
prevent the bot of doing anything.
include
One standalone value, either the name of a dictionary in your
- file or the name of a callable function that returns such a
- dictionary. This dictionary may have any of the five above keys
- (but not 'include' itself!), and the lists belonging to those
- keys will be added to your exceptions. This way you may define
- one or more basic collection of exceptions used for multiple
- fixes, and add separate exceptions to each fix.
+ file or the name of a callable function that takes the name of
+ the fix as argument and returns a dictionary of exceptions.
+ This dictionary may have any of the five above keys (but not
+ 'include' itself!), and the lists belonging to those keys will
+ be added to your exceptions. This way you may define one or
+ more basic collection of exceptions used for multiple fixes,
+ and add separate exceptions to each fix.
"""
self.generator = generator
@@ -811,6 +812,7 @@
else:
# Perform one of the predefined actions.
+ fixname = fix # Save the name for passing to exceptions function.
try:
fix = fixes.fixes[fix]
except KeyError:
@@ -833,7 +835,7 @@
if 'include' in exceptions:
incl = exceptions['include']
if callable(incl):
- baseExcDict = incl()
+ baseExcDict = incl(fixname)
else:
try:
baseExcDict = incl
http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9922
Revision: 9922
Author: binbot
Date: 2012-02-22 17:26:32 +0000 (Wed, 22 Feb 2012)
Log Message:
-----------
Enable use of one or more basic exceptions dictionary for multiple fixes.
Use the 'include' key in your fix to give a dictionary or function with further exceptions that will be added to the exceptions defined in the fix.
Prevents user of copying exceptions all the time and modifying them in several fixes collaterally.
Modified Paths:
--------------
trunk/pywikipedia/replace.py
Modified: trunk/pywikipedia/replace.py
===================================================================
--- trunk/pywikipedia/replace.py 2012-02-21 19:28:18 UTC (rev 9921)
+++ trunk/pywikipedia/replace.py 2012-02-22 17:26:32 UTC (rev 9922)
@@ -94,8 +94,8 @@
Default value is 60. Ignored when reading an XML file.
-fix:XYZ Perform one of the predefined replacements tasks, which are
- given in the dictionary 'fixes' defined inside the file
- fixes.py.
+ given in the dictionary 'fixes' defined inside the files
+ fixes.py and user-fixes.py.
The -regex, -recursive and -nocase argument and given
replacements and exceptions will be ignored if you use -fix
and they are present in the 'fixes' dictionary.
@@ -315,6 +315,14 @@
This is not an exception, and is here for technical reasons.
Listing the same regex in title and require-title will thus
prevent the bot of doing anything.
+ include
+ One standalone value, either the name of a dictionary in your
+ file or the name of a callable function that returns such a
+ dictionary. This dictionary may have any of the five above keys
+ (but not 'include' itself!), and the lists belonging to those
+ keys will be added to your exceptions. This way you may define
+ one or more basic collection of exceptions used for multiple
+ fixes, and add separate exceptions to each fix.
"""
self.generator = generator
@@ -820,6 +828,26 @@
fix['msg'])
if "exceptions" in fix:
exceptions = fix['exceptions']
+ # Try to append common extensions for multiple fixes.
+ # It must be either a dictionary or a function that returns a dict.
+ if 'include' in exceptions:
+ incl = exceptions['include']
+ if callable(incl):
+ baseExcDict = incl()
+ else:
+ try:
+ baseExcDict = incl
+ except NameError:
+ pywikibot.output(
+ u'\nIncluded exceptions dictionary does not exist.' +
+ u' Continuing with the exceptions\ngiven in fix.\n')
+ baseExcDict = None
+ if baseExcDict:
+ for l in baseExcDict:
+ try:
+ exceptions[l].extend(baseExcDict[l])
+ except KeyError:
+ exceptions[l] = baseExcDict[l]
if "recursive" in fix:
recursive = fix['recursive']
if "nocase" in fix:
http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9921
Revision: 9921
Author: binbot
Date: 2012-02-21 19:28:18 +0000 (Tue, 21 Feb 2012)
Log Message:
-----------
Some code combing, PEP 8, nicer output even for non-English replacements, error handling. Nothing serious this time. :-)
Modified Paths:
--------------
trunk/pywikipedia/replace.py
Modified: trunk/pywikipedia/replace.py
===================================================================
--- trunk/pywikipedia/replace.py 2012-02-21 14:35:09 UTC (rev 9920)
+++ trunk/pywikipedia/replace.py 2012-02-21 19:28:18 UTC (rev 9921)
@@ -230,7 +230,8 @@
and not self.isTextExcepted(entry.text):
new_text = entry.text
for old, new in self.replacements:
- new_text = pywikibot.replaceExcept(new_text, old, new, self.excsInside, self.site)
+ new_text = pywikibot.replaceExcept(
+ new_text, old, new, self.excsInside, self.site)
if new_text != entry.text:
yield pywikibot.Page(self.site, entry.title)
except KeyboardInterrupt:
@@ -665,7 +666,7 @@
excappend = False
if len(arg) == 11:
excoutfilename = pywikibot.input(
- u'Please enter the filename to save the excepted titles' +\
+ u'Please enter the filename to save the excepted titles' +
u'\n(will be deleted if exists):')
else:
excoutfilename = arg[12:]
@@ -679,7 +680,7 @@
append = False
if len(arg) == 8:
filename = pywikibot.input(
- u'Please enter the filename to save the titles' + \
+ u'Please enter the filename to save the titles' +
u'\n(will be deleted if exists):')
else:
filename = arg[9:]
@@ -692,10 +693,16 @@
elif arg.startswith('-replacementfile'):
if len(arg) == len('-replacementfile'):
replacefile = pywikibot.input(
-u"""Please enter the filename to read replacements from:""")
+ u'Please enter the filename to read replacements from:')
else:
replacefile = arg[len('-replacementfile')+1:]
- commandline_replacements.extend([x.lstrip(u'\uFEFF').rstrip('\r\n') for x in codecs.open(replacefile, 'r', 'utf-8')])
+ try:
+ commandline_replacements.extend(
+ [x.lstrip(u'\uFEFF').rstrip('\r\n')
+ for x in codecs.open(replacefile, 'r', 'utf-8')])
+ except IOError:
+ raise pywikibot.Error(
+ '\n%s cannot be opened. Try again :-)' % replacefile)
elif arg.startswith('-excepttitle:'):
exceptions['title'].append(arg[13:])
elif arg.startswith('-requiretitle:'):
@@ -734,7 +741,8 @@
commandline_replacements.append(arg)
if pywikibot.verbose:
- pywikibot.output(u"commandline_replacements: %r" % commandline_replacements)
+ pywikibot.output(u"commandline_replacements: " +
+ ', '.join(commandline_replacements))
if (len(commandline_replacements) % 2):
raise pywikibot.Error, 'require even number of replacements.'
@@ -772,7 +780,8 @@
replacements.append((old, new))
while True:
old = pywikibot.input(
-u'Please enter another text that should be replaced, or press Enter to start:')
+ u'Please enter another text that should be replaced,' +
+ u'\nor press Enter to start:')
if old == '':
change += ')'
break
@@ -786,7 +795,7 @@
pywikibot.output(u'The summary message will default to: %s'
% default_summary_message)
summary_message = pywikibot.input(
- u'Press Enter to use this default message, or enter a' + \
+ u'Press Enter to use this default message, or enter a ' +
u'description of the\nchanges your bot will make:')
if summary_message == '':
summary_message = default_summary_message
@@ -818,7 +827,8 @@
try:
replacements = fix['replacements']
except KeyError:
- pywikibot.output(u"No replacements given in fix, don't joke with me!")
+ pywikibot.output(
+ u"No replacements given in fix, don't joke with me!")
return
# Set the regular expression flags
@@ -838,7 +848,8 @@
oldR = re.compile(old, flags)
replacements[i] = oldR, new
- for exceptionCategory in ['title', 'require-title', 'text-contains', 'inside']:
+ for exceptionCategory in [
+ 'title', 'require-title', 'text-contains', 'inside']:
if exceptionCategory in exceptions:
patterns = exceptions[exceptionCategory]
if not regex:
http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9918
Revision: 9918
Author: binbot
Date: 2012-02-21 11:39:41 +0000 (Tue, 21 Feb 2012)
Log Message:
-----------
My old dream completed: with -saveexc or -saveexcnew you may save excepted titles for later use, and don't have to bother them again on next runs.
Currently you may paste the titles into exceptions of a fix, reading them from file will be implemented later.
+ some PEP 8
Modified Paths:
--------------
trunk/pywikipedia/replace.py
Modified: trunk/pywikipedia/replace.py
===================================================================
--- trunk/pywikipedia/replace.py 2012-02-21 00:07:55 UTC (rev 9917)
+++ trunk/pywikipedia/replace.py 2012-02-21 11:39:41 UTC (rev 9918)
@@ -43,6 +43,22 @@
-savenew Just like -save, except that overwrites the existing file.
Argument may also be given as "-savenew:filename".
+-saveexc With this parameter a new option will appear in choices:
+ "no+eXcept". If you press x, the text will not be replaced,
+ and the title of page will be saved to the given exception
+ file to exclude this page from future replacements. At the
+ moment you may paste the contents directly into 'title' list
+ of the exceptions dictionary of your fix (use tab to indent).
+ Reading back the list from file will be implemented later.
+ Argument may also be given as "-saveexc:filename".
+ Opens the file for append, if exists.
+
+-saveexcnew Just like -saveexc, except that overwrites the existing file.
+ Argument may also be given as "-saveexcnew:filename".
+
+-readexc Reserved for reading saved exceptions from a file.
+ Not implemented yet.
+
-addcat:cat_name Adds "cat_name" category to every altered page.
-excepttitle:XYZ Skip pages with titles that contain XYZ. If the -regex
@@ -252,7 +268,8 @@
"""
def __init__(self, generator, replacements, exceptions={},
acceptall=False, allowoverlap=False, recursive=False,
- addedCat=None, sleep=None, editSummary='', articles=None):
+ addedCat=None, sleep=None, editSummary='', articles=None,
+ exctitles=None):
"""
Arguments:
* generator - A generator that yields Page objects.
@@ -269,6 +286,10 @@
touched.
* articles - An open file to save the page titles. If None,
we work on our wikisite immediately (default).
+ Corresponds to titlefile variable of main().
+ * exctitles - An open file to save the excepted titles. If None,
+ we don't ask the user about saving them (default).
+ Corresponds to excoutfile variable of main().
Structure of the exceptions dictionary:
This dictionary can have these keys:
@@ -308,10 +329,13 @@
# Some function to set default editSummary should probably be added
self.editSummary = editSummary
self.articles = articles
+ self.exctitles = exctitles
- #An edit counter to split the file by 100 titles if -save or -savenew
- #is on, and to display the number of edited articles otherwise.
+ # An edit counter to split the file by 100 titles if -save or -savenew
+ # is on, and to display the number of edited articles otherwise.
self.editcounter = 0
+ # A counter for saved exceptions
+ self.exceptcounter = 0
def isTitleExcepted(self, title):
"""
@@ -369,6 +393,16 @@
(lambda x: bool(x-1) and 's were' or ' was')
(self.editcounter)))
+ def writeExceptCounter(self):
+ """ This writes the counter of saved exceptions if applicable. """
+ if self.exctitles:
+ pywikibot.output(u'%d exception%s saved.'
+ % (self.exceptcounter,
+ (lambda x: bool(x-1) and 's were' or ' was')
+ (self.exceptcounter)))
+ else:
+ print 888
+
def splitLine(self):
"""Returns a splitline after every 100th title. Splitline is in HTML
comment format in case we want to insert the list into a wikipage.
@@ -433,8 +467,15 @@
pywikibot.showDiff(original_text, new_text)
if self.acceptall:
break
- choice = pywikibot.inputChoice(
+ if self.exctitles:
+ choice = pywikibot.inputChoice(
u'Do you want to accept these changes?',
+ ['Yes', 'No', 'no+eXcept', 'Edit',
+ 'open in Browser', 'All', 'Quit'],
+ ['y', 'N', 'x', 'e', 'b', 'a', 'q'], 'N')
+ else:
+ choice = pywikibot.inputChoice(
+ u'Do you want to accept these changes?',
['Yes', 'No', 'Edit', 'open in Browser', 'All',
'Quit'],
['y', 'N', 'e', 'b', 'a', 'q'], 'N')
@@ -461,17 +502,22 @@
continue
if choice == 'q':
self.writeEditCounter()
+ self.writeExceptCounter()
return
if choice == 'a':
self.acceptall = True
+ if choice == 'x': #May happen only if self.exctitles isn't None
+ self.exctitles.write(u"u'%s',\n" % page.title())
+ self.exctitles.flush()
+ self.exceptcounter += 1
if choice == 'y':
if not self.articles:
- #Primary behaviour: working on wiki
+ # Primary behaviour: working on wiki
page.put_async(new_text, self.editSummary)
self.editcounter += 1
- #Bug: this increments even if put_async fails
- #This is separately in two clauses of if for
- #future purposes to get feedback form put_async
+ # Bug: this increments even if put_async fails
+ # This is separately in two clauses of if for
+ # future purposes to get feedback form put_async
else:
#Save the title for later processing instead of editing
self.editcounter += 1
@@ -510,6 +556,7 @@
#Finally:
self.writeEditCounter()
+ self.writeExceptCounter()
def prepareRegexForMySQL(pattern):
pattern = pattern.replace('\s', '[:space:]')
@@ -579,11 +626,18 @@
# too much CPU
sleep = None
# Do not save the page titles, rather work on wiki
- titlefile = None
- filename = None
+ filename = None # The name of the file to save titles
+ titlefile = None # The file object itself
# If we save, primary behaviour is append rather then new file
append = True
+ # Default: don't write titles to exception file and don't read them.
+ excoutfilename = None # The name of the file to save exceptions
+ excoutfile = None # The file object itself
+ # excinfilename: reserved for later use (reading back exceptions)
+ # If we save exceptions, primary behaviour is append
+ excappend = True
+
# Read commandline parameters.
for arg in pywikibot.handleArgs(*args):
if arg == '-regex':
@@ -608,11 +662,26 @@
u'Which page do you want to change?'))
else:
PageTitles.append(arg[6:])
+ elif arg.startswith('-saveexcnew'):
+ excappend = False
+ if len(arg) == 11:
+ excoutfilename = pywikibot.input(
+ u'Please enter the filename to save the excepted titles' +\
+ u'\n(will be deleted if exists):')
+ else:
+ excoutfilename = arg[12:]
+ elif arg.startswith('-saveexc'):
+ if len(arg) == 8:
+ excoutfilename = pywikibot.input(
+ u'Please enter the filename to save the excepted titles:')
+ else:
+ excoutfilename = arg[9:]
elif arg.startswith('-savenew'):
append = False
if len(arg) == 8:
filename = pywikibot.input(
-u'Please enter the filename to save the titles \n(will be deleted if exists):')
+ u'Please enter the filename to save the titles' + \
+ u'\n(will be deleted if exists):')
else:
filename = arg[9:]
elif arg.startswith('-save'):
@@ -718,7 +787,8 @@
pywikibot.output(u'The summary message will default to: %s'
% default_summary_message)
summary_message = pywikibot.input(
-u'Press Enter to use this default message, or enter a description of the\nchanges your bot will make:')
+ u'Press Enter to use this default message, or enter a' + \
+ u'description of the\nchanges your bot will make:')
if summary_message == '':
summary_message = default_summary_message
editSummary = summary_message
@@ -752,7 +822,7 @@
pywikibot.output(u"No replacements given in fix, don't joke with me!")
return
- #Set the regular expression flags
+ # Set the regular expression flags
flags = re.UNICODE
if caseInsensitive:
flags = flags | re.IGNORECASE
@@ -816,27 +886,38 @@
preloadingGen = pagegenerators.PreloadingGenerator(gen,
pageNumber=maxquerysize)
- #Finally we open the file for page titles or set article to None
+ # Finally we open the file for page titles or set parameter article to None
if filename:
try:
- #This opens in strict error mode, that means bot will stop
- #on encoding errors with ValueError.
- #See http://docs.python.org/library/codecs.html#codecs.open
+ # This opens in strict error mode, that means bot will stop
+ # on encoding errors with ValueError.
+ # See http://docs.python.org/library/codecs.html#codecs.open
titlefile = codecs.open(filename, encoding='utf-8',
mode=(lambda x: x and 'a' or 'w')(append))
except IOError:
pywikibot.output("%s cannot be opened for writing." %
filename)
return
+ # The same process with exceptions file:
+ if excoutfilename:
+ try:
+ excoutfile = codecs.open(
+ excoutfilename, encoding='utf-8',
+ mode=(lambda x: x and 'a' or 'w')(excappend))
+ except IOError:
+ pywikibot.output("%s cannot be opened for writing." %
+ excoutfilename)
+ return
bot = ReplaceRobot(preloadingGen, replacements, exceptions, acceptall,
allowoverlap, recursive, add_cat, sleep, editSummary,
- titlefile)
+ titlefile, excoutfile)
try:
bot.run()
finally:
if titlefile:
- #Just for the spirit of programming (it was flushed)
+ # Just for the spirit of programming (they were flushed)
titlefile.close()
+ excoutfile.close()
if __name__ == "__main__":
http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9917
Revision: 9917
Author: binbot
Date: 2012-02-21 00:07:55 +0000 (Tue, 21 Feb 2012)
Log Message:
-----------
Some documentation of require-title
Modified Paths:
--------------
trunk/pywikipedia/replace.py
Modified: trunk/pywikipedia/replace.py
===================================================================
--- trunk/pywikipedia/replace.py 2012-02-20 15:54:09 UTC (rev 9916)
+++ trunk/pywikipedia/replace.py 2012-02-21 00:07:55 UTC (rev 9917)
@@ -145,7 +145,7 @@
"""
from __future__ import generators
#
-# (C) Daniel Herding & the Pywikipedia team, 2004-2011
+# (C) Daniel Herding & the Pywikipedia team, 2004-2012
#
__version__='$Id$'
#
@@ -287,6 +287,12 @@
inside-tags
A list of strings. These strings must be keys from the
exceptionRegexes dictionary in pywikibot.replaceExcept().
+ require-title
+ Opposite of title. Only pages with titles that are matched by
+ ALL of these regular expressions will be processed.
+ This is not an exception, and is here for technical reasons.
+ Listing the same regex in title and require-title will thus
+ prevent the bot of doing anything.
"""
self.generator = generator
http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9916
Revision: 9916
Author: drtrigon
Date: 2012-02-20 15:54:09 +0000 (Mon, 20 Feb 2012)
Log Message:
-----------
follow-up to r9915 in order to add message documentation in qqq
Modified Paths:
--------------
branches/rewrite/scripts/i18n/thirdparty.py
Modified: branches/rewrite/scripts/i18n/thirdparty.py
===================================================================
--- branches/rewrite/scripts/i18n/thirdparty.py 2012-02-20 15:13:48 UTC (rev 9915)
+++ branches/rewrite/scripts/i18n/thirdparty.py 2012-02-20 15:54:09 UTC (rev 9916)
@@ -184,7 +184,7 @@
'thirdparty-drtrigonbot-sum_disc-summary-clean': u'Discussion summary added with clean up: %(num)i entries',
'thirdparty-drtrigonbot-sum_disc-summary-head': u'robot',
'thirdparty-drtrigonbot-sum_disc-summary-mod': u'Discussion summary updated: %(num)i entries in %(page)s',
- 'thirdparty-drtrigonbot-subster-summary-mod': u'substituting %(tags)s tag(s).',
+ 'thirdparty-drtrigonbot-subster-summary-mod': u'substituting %(tags)s tag(s)',
},
# Author: Vivaelcelta
'es': {
@@ -698,6 +698,7 @@
'thirdparty-drtrigonbot-sum_disc-summary-clean': u'Summary line second part if there are additional results and old get removed.',
'thirdparty-drtrigonbot-sum_disc-summary-head': u'Summary line first part.',
'thirdparty-drtrigonbot-sum_disc-summary-mod': u'Summary line second part if the page had to be modified due to new results. Parameters:\n* %(num)i is the numer of entries\n* %(page)s is the page that was updated',
+ 'thirdparty-drtrigonbot-subster-summary-mod': u'Summary line (second part) if the page had to be modified due to new results. Parameters:\n* %(tags)s is a list of tag(s) updated on page',
},
# Author: DrTrigon
# Author: Rubin
http://www.mediawiki.org/wiki/Special:Code/pywikipedia/9914
Revision: 9914
Author: binbot
Date: 2012-02-20 09:32:00 +0000 (Mon, 20 Feb 2012)
Log Message:
-----------
Enable calling main() of arbitrary module from within another script.
Solution proposed by Merlijn, http://thread.gmane.org/gmane.comp.python.pywikipediabot.general/12420/focu…
Modified Paths:
--------------
trunk/pywikipedia/wikipedia.py
Modified: trunk/pywikipedia/wikipedia.py
===================================================================
--- trunk/pywikipedia/wikipedia.py 2012-02-20 08:52:32 UTC (rev 9913)
+++ trunk/pywikipedia/wikipedia.py 2012-02-20 09:32:00 UTC (rev 9914)
@@ -7688,6 +7688,10 @@
return os.path.basename(called)
def _decodeArg(arg):
+ # We may pass a Unicode string to a script upon importing and calling
+ # main() from another script.
+ if isinstance(arg,unicode):
+ return arg
if sys.platform == 'win32':
if config.console_encoding in ('cp437', 'cp850'):
# Western Windows versions give parameters encoded as windows-1252