Revision: 6270 Author: cydeweys Date: 2009-01-19 15:50:18 +0000 (Mon, 19 Jan 2009)
Log Message: ----------- Changing the GeneratorFactory so that instead of handleArg() being called once and returning a generator, it is called many times to accumulate a list of pages to work on, and then, at the end, the combined page generator is explicitly asked for. This makes the GeneratorFactory a lot more useful for a wide variety of use cases.
Also, GeneratorFactory now understands the -page parameter, so some duplicate code was removed from bots that handled this individually.
Not in this commit, but coming soon, will be the handling of -namespace in the GeneratorFactory. This will remove quite a bit of duplicate code from individual bots and will make most bots more flexible.
Modified Paths: -------------- trunk/pywikipedia/add_text.py trunk/pywikipedia/basic.py trunk/pywikipedia/blockpageschecker.py trunk/pywikipedia/category.py trunk/pywikipedia/commonscat.py trunk/pywikipedia/copyright.py trunk/pywikipedia/cosmetic_changes.py trunk/pywikipedia/fixing_redirects.py trunk/pywikipedia/imagecopy.py trunk/pywikipedia/imagerecat.py trunk/pywikipedia/imageuncat.py trunk/pywikipedia/inline_images.py trunk/pywikipedia/interwiki.py trunk/pywikipedia/isbn.py trunk/pywikipedia/lonelypages.py trunk/pywikipedia/movepages.py trunk/pywikipedia/noreferences.py trunk/pywikipedia/pagegenerators.py trunk/pywikipedia/piper.py trunk/pywikipedia/reflinks.py trunk/pywikipedia/replace.py trunk/pywikipedia/selflink.py trunk/pywikipedia/table2wiki.py trunk/pywikipedia/template.py trunk/pywikipedia/weblinkchecker.py
Modified: trunk/pywikipedia/add_text.py =================================================================== --- trunk/pywikipedia/add_text.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/add_text.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -283,7 +283,10 @@ elif arg == '-always': always = True else: - generator = genFactory.handleArg(arg) + genFactory.handleArg(arg) + + if not generator: + generator = genFactory.getCombinedGenerator() # Check if there are the minimal settings if not generator: raise NoEnoughData('You have to specify the generator you want to use for the script!')
Modified: trunk/pywikipedia/basic.py =================================================================== --- trunk/pywikipedia/basic.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/basic.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -119,10 +119,7 @@ else: # check if a standard argument like # -start:XYZ or -ref:Asdf was given. - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): pageTitleParts.append(arg)
if pageTitleParts != []: @@ -131,6 +128,8 @@ page = wikipedia.Page(wikipedia.getSite(), pageTitle) gen = iter([page])
+ if not gen: + gen = genFactory.getCombinedGenerator() if gen: # The preloading generator is responsible for downloading multiple # pages from the wiki simultaneously.
Modified: trunk/pywikipedia/blockpageschecker.py =================================================================== --- trunk/pywikipedia/blockpageschecker.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/blockpageschecker.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -228,7 +228,7 @@ else: generator = [wikipedia.Page(wikipedia.getSite(), arg[6:])] else: - generator = genFactory.handleArg(arg) + genFactory.handleArg(arg) # Take the right templates to use, the category and the comment TSP = wikipedia.translate(site, templateSemiProtection) TTP = wikipedia.translate(site, templateTotalProtection) @@ -239,6 +239,8 @@ category = wikipedia.translate(site, categoryToCheck) commentUsed = wikipedia.translate(site, comment) if not generator: + gen = genFactory.getCombinedGenerator() + if not generator: generator = list() wikipedia.output(u'Loading categories...') # Define the category if no other generator has been setted
Modified: trunk/pywikipedia/category.py =================================================================== --- trunk/pywikipedia/category.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/category.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -946,14 +946,19 @@ elif arg == '-recurse': recurse = True else: - gen = genFactory.handleArg(arg) + genFactory.handleArg(arg)
if action == 'add': + # Note that the add functionality is the only bot that actually uses the + # the generator factory. Every other bot creates its own generator exclusively + # from the command-line arguments that category.py understands. if not gen: - gen = genFactory.handleArg('-links') #default for backwords compatibility - # The preloading generator is responsible for downloading multiple - # pages from the wiki simultaneously. - gen = pagegenerators.PreloadingGenerator(gen) + gen = genFactory.getCombinedGenerator() + if not gen: + genFactory.handleArg('-links') #default for backwords compatibility + # The preloading generator is responsible for downloading multiple + # pages from the wiki simultaneously. + gen = pagegenerators.PreloadingGenerator(genFactory.getCombinedGenerator()) add_category(sort_by_last_name) elif action == 'remove': if (fromGiven == False):
Modified: trunk/pywikipedia/commonscat.py =================================================================== --- trunk/pywikipedia/commonscat.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/commonscat.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -345,8 +345,11 @@ elif arg == '-always': always = True else: - generator = genFactory.handleArg(arg) + genFactory.handleArg(arg) + if not generator: + generator = genFactory.getCombinedGenerator() + if not generator: raise add_text.NoEnoughData('You have to specify the generator you want to use for the script!')
pregenerator = pagegenerators.PreloadingGenerator(generator)
Modified: trunk/pywikipedia/copyright.py =================================================================== --- trunk/pywikipedia/copyright.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/copyright.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -1145,9 +1145,7 @@ if number < pageNumber: pageNumber = number else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator + genFactory.handleArg(arg)
if PageTitles: pages = [wikipedia.Page(wikipedia.getSite(), PageTitle) for PageTitle in PageTitles] @@ -1160,6 +1158,8 @@ if ids: checks_by_ids(ids)
+ if not gen: + gen = genFactory.getCombinedGenerator() if not gen and not ids and not text: # syntax error, show help text from the top of this file wikipedia.output(__doc__, 'utf-8')
Modified: trunk/pywikipedia/cosmetic_changes.py =================================================================== --- trunk/pywikipedia/cosmetic_changes.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/cosmetic_changes.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -393,10 +393,7 @@ genFactory = pagegenerators.GeneratorFactory()
for arg in wikipedia.handleArgs(): - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): pageTitle.append(arg)
# Disabled this check. Although the point is still valid, there @@ -412,6 +409,8 @@ page = wikipedia.Page(wikipedia.getSite(), ' '.join(pageTitle)) gen = iter([page]) if not gen: + gen = genFactory.getCombinedGenerator() + if not gen: wikipedia.showHelp() elif wikipedia.inputChoice(warning + '\nDo you really want to continue?', ['yes', 'no'], ['y', 'N'], 'N') == 'y': preloadingGen = pagegenerators.PreloadingGenerator(gen)
Modified: trunk/pywikipedia/fixing_redirects.py =================================================================== --- trunk/pywikipedia/fixing_redirects.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/fixing_redirects.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -181,10 +181,10 @@ else: namespace = int(arg[11:]) else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator + genFactory.handleArg(arg)
+ gen = genFactory.getCombinedGenerator() + mysite = wikipedia.getSite() if mysite.sitename() == 'wikipedia:nl': wikipedia.output(u'\03{lightred}There is consensus on the Dutch Wikipedia that bots should not be used to fix redirects.\03{default}')
Modified: trunk/pywikipedia/imagecopy.py =================================================================== --- trunk/pywikipedia/imagecopy.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/imagecopy.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -440,15 +440,12 @@ genFactory = pagegenerators.GeneratorFactory()
for arg in wikipedia.handleArgs(): - if arg.startswith('-page'): - if len(arg) == 5: - generator = [wikipedia.Page(wikipedia.getSite(), wikipedia.input(u'What page do you want to use?'))] - else: - generator = [wikipedia.Page(wikipedia.getSite(), arg[6:])] - elif arg == '-always': + if arg == '-always': always = True else: - generator = genFactory.handleArg(arg) + genFactory.handleArg(arg) + + generator = genFactory.getCombinedGenerator() if not generator: raise add_text.NoEnoughData('You have to specify the generator you want to use for the script!')
Modified: trunk/pywikipedia/imagerecat.py =================================================================== --- trunk/pywikipedia/imagerecat.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/imagerecat.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -313,15 +313,12 @@ site = wikipedia.getSite(u'commons', u'commons') wikipedia.setSite(site) for arg in wikipedia.handleArgs(): - if arg.startswith('-page'): - if len(arg) == 5: - generator = [wikipedia.Page(site, wikipedia.input(u'What page do you want to use?'))] - else: - generator = [wikipedia.Page(site, arg[6:])] - elif arg == '-onlyfilter': + if arg == '-onlyfilter': onlyfilter = True else: - generator = genFactory.handleArg(arg) + genFactory.handleArg(arg) + + gen = genFactory.getCombinedGenerator() if not generator: generator = pagegenerators.CategorizedPageGenerator(catlib.Category(site, u'Category:Media needing categories'), recurse=True)
Modified: trunk/pywikipedia/imageuncat.py =================================================================== --- trunk/pywikipedia/imageuncat.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/imageuncat.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -1339,18 +1339,16 @@ site = wikipedia.getSite(u'commons', u'commons') wikipedia.setSite(site) for arg in wikipedia.handleArgs(): - if arg.startswith('-page'): - if len(arg) == 5: - generator = [wikipedia.Page(site, wikipedia.input(u'What page do you want to use?'))] - else: - generator = [wikipedia.Page(site, arg[6:])] - elif arg.startswith('-yesterday'): + if arg.startswith('-yesterday'): generator = uploadedYesterday(site) elif arg.startswith('-recentchanges'): generator = recentChanges(site) else: - generator = genFactory.handleArg(arg) + genFactory.handleArg(arg) + if not generator: + generator = genFactory.getCombinedGenerator() + if not generator: wikipedia.output('You have to specify the generator you want to use for the program!') else: pregenerator = pagegenerators.PreloadingGenerator(generator)
Modified: trunk/pywikipedia/inline_images.py =================================================================== --- trunk/pywikipedia/inline_images.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/inline_images.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -88,10 +88,7 @@ genFactory = pagegenerators.GeneratorFactory()
for arg in wikipedia.handleArgs(): - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): pageTitle.append(arg)
if pageTitle: @@ -99,6 +96,8 @@ page = wikipedia.Page(wikipedia.getSite(), ' '.join(pageTitle)) gen = iter([page]) if not gen: + gen = genFactory.getCombinedGenerator() + if not gen: wikipedia.showHelp('inline_images') else: preloadingGen = pagegenerators.PreloadingGenerator(gen)
Modified: trunk/pywikipedia/interwiki.py =================================================================== --- trunk/pywikipedia/interwiki.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/interwiki.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -1726,12 +1726,10 @@ elif arg == '-back': globalvar.nobackonly = True else: - generator = genFactory.handleArg(arg) - if generator: - hintlessPageGen = generator - else: + if not genFactory.handleArg(arg): singlePageTitle.append(arg)
+ # ensure that we don't try to change main page try: site = wikipedia.getSite() @@ -1764,6 +1762,8 @@
bot = InterwikiBot()
+ if not hintlessPageGen: + hintlessPageGen = genFactory.getCombinedGenerator() if hintlessPageGen: if len(namespaces) > 0: hintlessPageGen = pagegenerators.NamespaceFilterPageGenerator(hintlessPageGen, namespaces)
Modified: trunk/pywikipedia/isbn.py =================================================================== --- trunk/pywikipedia/isbn.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/isbn.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -1473,16 +1473,15 @@ elif arg == '-format': format = True else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): pageTitle.append(arg)
if pageTitle: page = wikipedia.Page(wikipedia.getSite(), ' '.join(pageTitle)) gen = iter([page]) if not gen: + gen = genFactory.getCombinedGenerator() + if not gen: wikipedia.showHelp('isbn') else: if namespaces != []:
Modified: trunk/pywikipedia/lonelypages.py =================================================================== --- trunk/pywikipedia/lonelypages.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/lonelypages.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -134,19 +134,18 @@ nwlimit = int(arg[10:]) generator = wikipedia.getSite().newpages(number = nwlimit) nwpages = True - elif arg.startswith('-page'): - if len(arg) == 5: - generator = [wikipedia.Page(wikipedia.getSite(), wikipedia.input(u'How many pages do you want to check?'))] - else: - generator = [wikipedia.Page(wikipedia.getSite(), arg[6:])] elif arg == '-always': always = True else: - generator = genFactory.handleArg(arg) + genFactory.handleArg(arg) # Retrive the site wikiSite = wikipedia.getSite() + + if not generator: + generator = genFactory.getCombinedGenerator() + # If the generator is not given, use the default one - if generator == None: + if not generator: generator = wikiSite.lonelypages(repeat = True, number = limit) # Take the configurations according to our project comment = wikipedia.translate(wikiSite, commento)
Modified: trunk/pywikipedia/movepages.py =================================================================== --- trunk/pywikipedia/movepages.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/movepages.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -269,9 +269,7 @@ else: summary = arg[9:] else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator + genFactory.handleArg(arg)
if oldName: wikipedia.output(u'WARNING: -from:%s without -to:' % oldName) @@ -279,6 +277,9 @@ page = wikipedia.Page(wikipedia.getSite(), pair[0]) bot = MovePagesBot(None, prefix, delete, always, skipredirects, summary) bot.moveOne(page, pair[1]) + + if not gen: + gen = genFactory.getCombinedGenerator() if gen: preloadingGen = pagegenerators.PreloadingGenerator(gen) bot = MovePagesBot(preloadingGen, prefix, delete, always, skipredirects, summary)
Modified: trunk/pywikipedia/noreferences.py =================================================================== --- trunk/pywikipedia/noreferences.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/noreferences.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -453,16 +453,15 @@ elif arg == '-always': always = True else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): pageTitle.append(arg)
if pageTitle: page = wikipedia.Page(wikipedia.getSite(), ' '.join(pageTitle)) gen = iter([page]) if not gen: + gen = genFactory.getCombinedGenerator() + if not gen: wikipedia.showHelp('noreferences') else: if namespaces != []:
Modified: trunk/pywikipedia/pagegenerators.py =================================================================== --- trunk/pywikipedia/pagegenerators.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/pagegenerators.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -796,20 +796,32 @@ to work on. """ def __init__(self): - pass + self.gens = []
- def setCategoryGen(self, arg, length, recurse = False): + """ + This function returns the combination of all accumulated generators + that have been created in the process of handling arguments. + Only call it after all arguments have been parsed. + """ + def getCombinedGenerator(self): + if (len(self.gens) == 0): + return None + elif (len(self.gens) == 1): + return DuplicateFilterPageGenerator(self.gens[0]) + else: + return DuplicateFilterPageGenerator(CombinedPageGenerator(self.gens)) + + def getCategoryGen(self, arg, length, recurse = False): if len(arg) == length: categoryname = wikipedia.input(u'Please enter the category name:') else: categoryname = arg[length + 1:]
ind = categoryname.find('|') + startfrom = None if ind > 0: startfrom = categoryname[ind + 1:] categoryname = categoryname[:ind] - else: - startfrom = None
cat = catlib.Category(wikipedia.getSite(), 'Category:%s' % categoryname) return CategorizedPageGenerator(cat, start = startfrom, recurse = recurse) @@ -830,6 +842,14 @@ cat = catlib.Category(wikipedia.getSite(), 'Category:%s' % categoryname) return SubCategoriesPageGenerator(cat, start = startfrom, recurse = recurse)
+ """ + This function parses one argument at a time. If it is recognized as an + argument that specifies a generator, a generator is created and added + to the accumulation list, and the function returns true. Otherwise, it + returns false, so that callee can try parsing the argument. + Call getCombinedGenerator() after all arguments have been parsed to get + the final output generator. + """ def handleArg(self, arg): gen = None if arg.startswith('-filelinks'): @@ -872,16 +892,23 @@ textfilename = wikipedia.input(u'Please enter the local file name:') gen = TextfilePageGenerator(textfilename) elif arg.startswith('-catr'): - gen = self.setCategoryGen(arg, 5, recurse = True) + gen = self.getCategoryGen(arg, 5, recurse = True) elif arg.startswith('-cat'): - gen = self.setCategoryGen(arg, 4) + gen = self.getCategoryGen(arg, len('-cat')) + elif arg.startswith('-category'): + gen = self.getCategoryGen(arg, len('-category')) elif arg.startswith('-subcatsr'): gen = self.setSubCategoriesGen(arg, 9, recurse = True) elif arg.startswith('-subcats'): gen = self.setSubCategoriesGen(arg, 8) # This parameter is deprecated, catr should be used instead. elif arg.startswith('-subcat'): - gen = self.setCategoryGen(arg, 7, recurse = True) + gen = self.getCategoryGen(arg, 7, recurse = True) + elif arg.startswith('-page'): + if len(arg) == len('-page'): + gen = [wikipedia.Page(wikipedia.getSite(), wikipedia.input(u'What page do you want to use?'))] + else: + gen = [wikipedia.Page(wikipedia.getSite(), arg[len('-page:'):])] elif arg.startswith('-uncatfiles'): gen = UnCategorizedImageGenerator() elif arg.startswith('-uncatcat'): @@ -958,10 +985,12 @@ elif arg.startswith('-yahoo'): gen = YahooSearchPageGenerator(arg[7:]) else: - return None - # make sure all yielded pages are unique - gen = DuplicateFilterPageGenerator(gen) - return gen + pass + if gen: + self.gens.append(gen) + return True + else: + return False
if __name__ == "__main__": try:
Modified: trunk/pywikipedia/piper.py =================================================================== --- trunk/pywikipedia/piper.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/piper.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -188,10 +188,7 @@ else: # check if a standard argument like # -start:XYZ or -ref:Asdf was given. - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): pageTitleParts.append(arg)
if pageTitleParts != []: @@ -200,6 +197,8 @@ page = wikipedia.Page(wikipedia.getSite(), pageTitle) gen = iter([page])
+ if not gen: + gen = genFactory.getCombinedGenerator() if gen: # The preloading generator is responsible for downloading multiple # pages from the wiki simultaneously.
Modified: trunk/pywikipedia/reflinks.py =================================================================== --- trunk/pywikipedia/reflinks.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/reflinks.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -678,9 +678,7 @@ namespaces = [] generator = None for arg in wikipedia.handleArgs(): - if arg.startswith('-page:'): - PageTitles.append(arg[6:]) - elif arg.startswith('-namespace:'): + if arg.startswith('-namespace:'): try: namespaces.append(int(arg[11:])) except ValueError: @@ -706,7 +704,7 @@ else: xmlFilename = arg[5:] else: - generator = genFactory.handleArg(arg) + genFactory.handleArg(arg)
if xmlFilename: try: @@ -714,10 +712,9 @@ except NameError: xmlStart = None generator = XmlDumpPageGenerator(xmlFilename, xmlStart, namespaces) - elif PageTitles: - pages= [wikipedia.Page(wikipedia.getSite(), PageTitle) for PageTitle in PageTitles] - generator = iter(pages) if not generator: + generator = genFactory.getCombinedGenerator() + if not generator: # syntax error, show help text from the top of this file wikipedia.showHelp('reflinks') return
Modified: trunk/pywikipedia/replace.py =================================================================== --- trunk/pywikipedia/replace.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/replace.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -16,10 +16,6 @@ Argument can also be given as "-page:pagetitle". You can give this parameter multiple times to edit multiple pages.
--category Works on all of the pages in a specific category. Specify - this argument multiple times to work on multiple categories - simultaneously. - Furthermore, the following command line parameters are supported:
-regex Make replacements using regular expressions. If this argument @@ -515,8 +511,6 @@ # Between a regex and another (using -fix) sleep some time (not to waste # too much CPU sleep = None - # A list of categories whose pages we should process. - categories = []
# Read commandline parameters. for arg in wikipedia.handleArgs(*args): @@ -536,8 +530,6 @@ xmlFilename = arg[5:] elif arg =='-sql': useSql = True - elif arg.startswith('-category'): - categories.append(arg[len('-category:'):]) elif arg.startswith('-page'): if len(arg) == 5: PageTitles.append(wikipedia.input( @@ -581,10 +573,7 @@ elif arg.startswith('-allowoverlap'): allowoverlap = True else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): commandline_replacements.append(arg)
if (len(commandline_replacements) % 2): @@ -704,15 +693,14 @@ %s LIMIT 200""" % (whereClause, exceptClause) gen = pagegenerators.MySQLPageGenerator(query) - elif categories: - gens = [pagegenerators.CategorizedPageGenerator(catlib.Category(wikipedia.getSite(), 'Category:' + t)) for t in categories] - gen = pagegenerators.DuplicateFilterPageGenerator(pagegenerators.CombinedPageGenerator(gens)) elif PageTitles: pages = [wikipedia.Page(wikipedia.getSite(), PageTitle) for PageTitle in PageTitles] gen = iter(pages)
if not gen: + gen = genFactory.getCombinedGenerator() + if not gen: # syntax error, show help text from the top of this file wikipedia.showHelp('replace') return
Modified: trunk/pywikipedia/selflink.py =================================================================== --- trunk/pywikipedia/selflink.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/selflink.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -224,16 +224,15 @@ except ValueError: namespaces.append(arg[11:]) else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): pageTitle.append(arg)
if pageTitle: page = wikipedia.Page(wikipedia.getSite(), ' '.join(pageTitle)) gen = iter([page]) if not gen: + gen = genFactory.getCombinedGenerator() + if not gen: wikipedia.showHelp('selflink') else: if namespaces != []:
Modified: trunk/pywikipedia/table2wiki.py =================================================================== --- trunk/pywikipedia/table2wiki.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/table2wiki.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -584,10 +584,7 @@ elif arg.startswith('-debug'): debug = True else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): page_title.append(arg)
# if the page is given as a command line argument, @@ -596,7 +593,10 @@ page_title = ' '.join(page_title) page = wikipedia.Page(wikipedia.getSite(), page_title) gen = iter([page]) + if not gen: + gen = genFactory.getCombinedGenerator() + if not gen: # show help wikipedia.showHelp('table2wiki') sys.exit(0)
Modified: trunk/pywikipedia/template.py =================================================================== --- trunk/pywikipedia/template.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/template.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -29,10 +29,6 @@
-always Don't bother asking to confirm any of the changes, Just Do It.
--page: Only edit a specific page. You can use this argument multiple times to work - on multiple pages. If the page title has spaces in it, enclose the entire - page name in quotes. - -category: Appends the given category to every page that is edited. This is useful when a category is being broken out from a template parameter or when templates are being upmerged but more information must be preserved. @@ -359,7 +355,7 @@ editSummary = '' addedCat = '' acceptAll = False - pageTitles = [] + genFactory = pagegenerators.GeneratorFactory() # If xmlfilename is None, references will be loaded from the live wiki. xmlfilename = None # read command line parameters @@ -384,13 +380,9 @@ editSummary = arg[len('-summary:'):] elif arg.startswith('-always'): acceptAll = True - elif arg.startswith('-page'): - if len(arg) == len('-page'): - pageTitles.append(wikipedia.input(u'Which page do you want to chage?')) - else: - pageTitles.append(arg[len('-page:'):]) else: - templateNames.append(arg) + if not genFactory.handleArg(arg): + templateNames.append(arg)
if subst or remove: for templateName in templateNames: @@ -411,10 +403,9 @@
if xmlfilename: gen = XmlDumpTemplatePageGenerator(oldTemplates, xmlfilename) - elif pageTitles: - pages = [wikipedia.Page(wikipedia.getSite(), pageTitle) for pageTitle in pageTitles] - gen = iter(pages) else: + gen = genFactory.getCombinedGenerator() + if not gen: gens = [] gens = [pagegenerators.ReferringPageGenerator(t, onlyTemplateInclusion = True) for t in oldTemplates] gen = pagegenerators.CombinedPageGenerator(gens)
Modified: trunk/pywikipedia/weblinkchecker.py =================================================================== --- trunk/pywikipedia/weblinkchecker.py 2009-01-19 12:53:43 UTC (rev 6269) +++ trunk/pywikipedia/weblinkchecker.py 2009-01-19 15:50:18 UTC (rev 6270) @@ -799,10 +799,7 @@ elif arg.startswith('-ignore:'): HTTPignore.append(int(arg[8:])) else: - generator = genFactory.handleArg(arg) - if generator: - gen = generator - else: + if not genFactory.handleArg(arg): singlePageTitle.append(arg)
if singlePageTitle: @@ -810,6 +807,8 @@ page = wikipedia.Page(wikipedia.getSite(), singlePageTitle) gen = iter([page])
+ if not gen: + gen = genFactory.getCombinedGenerator() if gen: if namespaces != []: gen = pagegenerators.NamespaceFilterPageGenerator(gen, namespaces)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
If this commit is causing any problems and you can't figure out what's wrong, please contact me and I'll have a look. Changing the behavior of the generator factory required touching every bot that used it, and I'm not familiar enough with some of the bots that I had to edit that I'm confidently I did it entirely right.
Oh yeah, and the upcoming -namespace parsing in GeneratorFactory is going to be really nifty, and should add a lot of functionality to bots that don't have a namespace filter already.
cydeweys@svn.wikimedia.org wrote:
Revision: 6270 Author: cydeweys Date: 2009-01-19 15:50:18 +0000 (Mon, 19 Jan 2009)
Log Message:
Changing the GeneratorFactory so that instead of handleArg() being called once and returning a generator, it is called many times to accumulate a list of pages to work on, and then, at the end, the combined page generator is explicitly asked for. This makes the GeneratorFactory a lot more useful for a wide variety of use cases.
Also, GeneratorFactory now understands the -page parameter, so some duplicate code was removed from bots that handled this individually.
Not in this commit, but coming soon, will be the handling of -namespace in the GeneratorFactory. This will remove quite a bit of duplicate code from individual bots and will make most bots more flexible.
Modified Paths:
trunk/pywikipedia/add_text.py trunk/pywikipedia/basic.py trunk/pywikipedia/blockpageschecker.py trunk/pywikipedia/category.py trunk/pywikipedia/commonscat.py trunk/pywikipedia/copyright.py trunk/pywikipedia/cosmetic_changes.py trunk/pywikipedia/fixing_redirects.py trunk/pywikipedia/imagecopy.py trunk/pywikipedia/imagerecat.py trunk/pywikipedia/imageuncat.py trunk/pywikipedia/inline_images.py trunk/pywikipedia/interwiki.py trunk/pywikipedia/isbn.py trunk/pywikipedia/lonelypages.py trunk/pywikipedia/movepages.py trunk/pywikipedia/noreferences.py trunk/pywikipedia/pagegenerators.py trunk/pywikipedia/piper.py trunk/pywikipedia/reflinks.py trunk/pywikipedia/replace.py trunk/pywikipedia/selflink.py trunk/pywikipedia/table2wiki.py trunk/pywikipedia/template.py trunk/pywikipedia/weblinkchecker.py
Hi Cyde,
Cyde Weys schreef:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
If this commit is causing any problems and you can't figure out what's wrong, please contact me and I'll have a look. Changing the behavior of the generator factory required touching every bot that used it, and I'm not familiar enough with some of the bots that I had to edit that I'm confidently I did it entirely right.
-cat: stopped working:
replace.py -cat:Wikipedia:Nog_te_categoriseren -regex "{{([xX]?)[nN]ocat}}" "{{xnocat||{{subst:CURRENTYEAR}}|{{subst:CURRENTMONTH}}|{{subst:CURRENTDAY}}}}" -summary:"Datum toevoegen aan nocat sjabloon" -always
Checked for running processes. 2 processes currently running, including the current process. Traceback (most recent call last): File "/home/bot/pywikipedia/replace.py", line 705, in ? main() File "/home/bot/pywikipedia/replace.py", line 562, in main if not genFactory.handleArg(arg): File "/home/bot/pywikipedia/pagegenerators.py", line 922, in handleArg gen = self.getCategoryGen(arg, len('-cat')) File "/home/bot/pywikipedia/pagegenerators.py", line 837, in getCategoryGen cat = catlib.Category(wikipedia.getSite(), categoryname) File "/home/bot/pywikipedia/catlib.py", line 67, in __init__ raise ValueError(u'BUG: %s is not in the category namespace!' % title) ValueError: BUG: Wikipedia:Nog_te_categoriseren is not in the category namespace!
And another one:
replace.py -cat:Wikipedia:Wikificatie_nodig -regex "{{([xX]?)[wW]ikify}}" "{{wikify||{{subst:CURRENTYEAR}}|{{subst:CURRENTMONTH}}|{{subst:CURRENTDAY}}}}" -summary:"Datum toevoegen aan wikify sjabloon" -always
Checked for running processes. 2 processes currently running, including the current process. Traceback (most recent call last): File "/home/bot/pywikipedia/replace.py", line 705, in ? main() File "/home/bot/pywikipedia/replace.py", line 562, in main if not genFactory.handleArg(arg): File "/home/bot/pywikipedia/pagegenerators.py", line 922, in handleArg gen = self.getCategoryGen(arg, len('-cat')) File "/home/bot/pywikipedia/pagegenerators.py", line 837, in getCategoryGen cat = catlib.Category(wikipedia.getSite(), categoryname) File "/home/bot/pywikipedia/catlib.py", line 67, in __init__ raise ValueError(u'BUG: %s is not in the category namespace!' % title) ValueError: BUG: Wikipedia:Wikificatie_nodig is not in the category namespace!
Could you please fix this?
Thank you,
Maarten
Ah, I see that the nlwiki has decided (for whatever reason) to have some categories whose names start with "Categorie:Wikipedia:Something something". Not a good idea, although technically valid names.
replace.py -cat:Categorie:Wikipedia:Nog_te_categoriseren (... etc.)
should work.
Russ
----- Original Message ----- From: "Maarten Dammers" maarten@mdammers.nl To: "Pywikipedia discussion list" pywikipedia-l@lists.wikimedia.org Sent: Thursday, January 22, 2009 9:44 AM Subject: Re: [Pywikipedia-l] SVN: [6270] trunk/pywikipedia
Hi Cyde,
Cyde Weys schreef:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
If this commit is causing any problems and you can't figure out what's wrong, please contact me and I'll have a look. Changing the behavior of the generator factory required touching every bot that used it, and I'm not familiar enough with some of the bots that I had to edit that I'm confidently I did it entirely right.
-cat: stopped working:
replace.py -cat:Wikipedia:Nog_te_categoriseren -regex "{{([xX]?)[nN]ocat}}" "{{xnocat||{{subst:CURRENTYEAR}}|{{subst:CURRENTMONTH}}|{{subst:CURRENTDAY}}}}" -summary:"Datum toevoegen aan nocat sjabloon" -always
Checked for running processes. 2 processes currently running, including the current process. Traceback (most recent call last): File "/home/bot/pywikipedia/replace.py", line 705, in ? main() File "/home/bot/pywikipedia/replace.py", line 562, in main if not genFactory.handleArg(arg): File "/home/bot/pywikipedia/pagegenerators.py", line 922, in handleArg gen = self.getCategoryGen(arg, len('-cat')) File "/home/bot/pywikipedia/pagegenerators.py", line 837, in getCategoryGen cat = catlib.Category(wikipedia.getSite(), categoryname) File "/home/bot/pywikipedia/catlib.py", line 67, in __init__ raise ValueError(u'BUG: %s is not in the category namespace!' % title) ValueError: BUG: Wikipedia:Nog_te_categoriseren is not in the category namespace!
And another one:
replace.py -cat:Wikipedia:Wikificatie_nodig -regex "{{([xX]?)[wW]ikify}}" "{{wikify||{{subst:CURRENTYEAR}}|{{subst:CURRENTMONTH}}|{{subst:CURRENTDAY}}}}" -summary:"Datum toevoegen aan wikify sjabloon" -always
Checked for running processes. 2 processes currently running, including the current process. Traceback (most recent call last): File "/home/bot/pywikipedia/replace.py", line 705, in ? main() File "/home/bot/pywikipedia/replace.py", line 562, in main if not genFactory.handleArg(arg): File "/home/bot/pywikipedia/pagegenerators.py", line 922, in handleArg gen = self.getCategoryGen(arg, len('-cat')) File "/home/bot/pywikipedia/pagegenerators.py", line 837, in getCategoryGen cat = catlib.Category(wikipedia.getSite(), categoryname) File "/home/bot/pywikipedia/catlib.py", line 67, in __init__ raise ValueError(u'BUG: %s is not in the category namespace!' % title) ValueError: BUG: Wikipedia:Wikificatie_nodig is not in the category namespace!
Could you please fix this?
Thank you,
Maarten
Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
Hi Russ,
Russell Blau schreef:
Ah, I see that the nlwiki has decided (for whatever reason) to have some categories whose names start with "Categorie:Wikipedia:Something something". Not a good idea, although technically valid names.
It is a good idea (don't mix up meta cats and topic cats) and actually policy at the nlwikipedia to prefix in this way: http://nl.wikipedia.org/w/index.php?title=Speciaal%3AVoorvoegselindex&fr...
replace.py -cat:Categorie:Wikipedia:Nog_te_categoriseren (... etc.)
should work.
And replace.py -cat:Wikipedia:Nog_te_categoriseren should work too
Maarten
Russ
I'll have to agree with marteen here, I think we should make the latter work too =)
2009/1/22 Maarten Dammers maarten@mdammers.nl:
Hi Russ,
Russell Blau schreef:
Ah, I see that the nlwiki has decided (for whatever reason) to have some categories whose names start with "Categorie:Wikipedia:Something something". Not a good idea, although technically valid names.
It is a good idea (don't mix up meta cats and topic cats) and actually policy at the nlwikipedia to prefix in this way: http://nl.wikipedia.org/w/index.php?title=Speciaal%3AVoorvoegselindex&fr...
replace.py -cat:Categorie:Wikipedia:Nog_te_categoriseren (... etc.)
should work.
And replace.py -cat:Wikipedia:Nog_te_categoriseren should work too
Maarten
Russ
Pywikipedia-l mailing list Pywikipedia-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l