Revision: 4239
Author: russblau
Date: 2007-09-11 15:47:20 +0000 (Tue, 11 Sep 2007)
Log Message:
-----------
+docu
Modified Paths:
--------------
trunk/pywikipedia/pagegenerators.py
Modified: trunk/pywikipedia/pagegenerators.py
===================================================================
--- trunk/pywikipedia/pagegenerators.py 2007-09-11 15:31:24 UTC (rev 4238)
+++ trunk/pywikipedia/pagegenerators.py 2007-09-11 15:47:20 UTC (rev 4239)
@@ -169,11 +169,14 @@
for page in allPages:
yield page
-def CategorizedPageGenerator(category, recurse = False, start = None):
+def CategorizedPageGenerator(category, recurse=False, start=None):
'''
Yields all pages in a specific category.
- If recurse is True, pages in subcategories are included as well.
+ If recurse is True, pages in subcategories are included as well; if
+ recurse is an int, only subcategories to that depth will be included
+ (e.g., recurse=2 will get pages in subcats and sub-subcats, but will
+ not go any further).
If start is a string value, only pages whose title comes after start
alphabetically are included.
'''
@@ -550,6 +553,12 @@
if pageNumber < 2:
raise ValueError("PreloadingGenerator needs to load more than 1 page.")
pagequeue = Queue.Queue(min(pageNumber//2, 10))
+ # Note: queue size will determine how quickly the Preloader goes back for
+ # more pages. If the queue size is unlimited, it will preload all pages
+ # before yielding any of them to the consumer. If the queue size is small,
+ # it will wait until most pages have been yielded before preloading the
+ # next batch. This value tries to strike a compromise, but may need
+ # adjustment based upon experience.
preloader = _Preloader(pagequeue, generator, pageNumber)
preloader.start()
while True:
Bugs item #1791651, was opened at 2007-09-10 17:03
Message generated for change (Settings changed) made by wikipedian
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1791651&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: imageReplace() is broken
Initial Comment:
Since wikipedian changed the function and tried to simplify the regexes, it introduced a bug. when attempting to replace an image in a gallery it will not remove old image instead it will just place the new image right before the old preventing any image from displaying. Image:foo.jpg is replaced by Image:bar.svg. in the gallery it becomes Image:bar.svgImage:foo.jpg
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1791651&group_…
Patches item #1792406, was opened at 2007-09-11 18:01
Message generated for change (Comment added) made by ebola_rulez
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1792406&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: Accepted
Priority: 5
Private: No
Submitted By: Aurimas Fischer (ebola_rulez)
Assigned to: Daniel Herding (wikipedian)
Summary: noreferences.py LT translation
Initial Comment:
Translation for lt.wiki
----------------------------------------------------------------------
>Comment By: Aurimas Fischer (ebola_rulez)
Date: 2007-09-11 18:38
Message:
Logged In: YES
user_id=959303
Originator: YES
I'm sure it is correct. We even have a template ({{Litref}}) that adds
<references /> into the 'Literatūra' section.
----------------------------------------------------------------------
Comment By: Daniel Herding (wikipedian)
Date: 2007-09-11 18:34
Message:
Logged In: YES
user_id=880694
Originator: NO
I applied it, but are you sure it is correct? Do you really want to add
<references/> tags into the 'Literature' section? I think 'Literatūra'
should be in the placeBeforeSections dictionary, where 'Nuorodos' (external
links) is. Is 'Šaltiniai' Lithuanian for 'References'?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1792406&group_…
Patches item #1792406, was opened at 2007-09-11 17:01
Message generated for change (Comment added) made by wikipedian
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1792406&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
>Resolution: Accepted
Priority: 5
Private: No
Submitted By: Aurimas Fischer (ebola_rulez)
Assigned to: Daniel Herding (wikipedian)
Summary: noreferences.py LT translation
Initial Comment:
Translation for lt.wiki
----------------------------------------------------------------------
>Comment By: Daniel Herding (wikipedian)
Date: 2007-09-11 17:34
Message:
Logged In: YES
user_id=880694
Originator: NO
I applied it, but are you sure it is correct? Do you really want to add
<references/> tags into the 'Literature' section? I think 'Literatūra'
should be in the placeBeforeSections dictionary, where 'Nuorodos' (external
links) is. Is 'Šaltiniai' Lithuanian for 'References'?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1792406&group_…
Revision: 4237
Author: wikipedian
Date: 2007-09-11 15:22:56 +0000 (Tue, 11 Sep 2007)
Log Message:
-----------
solved the problem where the bot doesn't know where to put the references section
Modified Paths:
--------------
trunk/pywikipedia/noreferences.py
Modified: trunk/pywikipedia/noreferences.py
===================================================================
--- trunk/pywikipedia/noreferences.py 2007-09-11 13:46:05 UTC (rev 4236)
+++ trunk/pywikipedia/noreferences.py 2007-09-11 15:22:56 UTC (rev 4237)
@@ -59,7 +59,8 @@
'de': [ # no explicit policy on where to put the references
u'Literatur',
u'Weblinks',
- u'Siehe auch'
+ u'Siehe auch',
+ u'Weblink', # bad, but common singular form of Weblinks
],
'en': [ # no explicit policy on where to put the references
u'Further reading',
@@ -170,7 +171,7 @@
oldText = page.get()
# Is there an existing section where we can add the references tag?
- for section in wikipedia.translate(wikipedia.getSite(), referencesSections):
+ for section in wikipedia.translate(page.site(), referencesSections):
sectionR = re.compile(r'\r\n=+ *%s *=+\r\n' % section)
index = 0
while index < len(oldText):
@@ -188,7 +189,7 @@
break
# Create a new section for the references tag
- for section in wikipedia.translate(wikipedia.getSite(), placeBeforeSections):
+ for section in wikipedia.translate(page.site(), placeBeforeSections):
# Find out where to place the new section
sectionR = re.compile(r'\r\n=+ *%s *=+\r\n' % section)
index = 0
@@ -200,16 +201,44 @@
index = match.end()
else:
wikipedia.output(u'Adding references section before %s section...\n' % section)
- pos = match.start()
- newSection = u'\n== %s ==\n\n<references/>\n' % wikipedia.translate(wikipedia.getSite(), referencesSections)[0]
- newText = oldText[:match.start()] + newSection + oldText[match.start():]
- self.save(page, newText)
+ index = match.start()
+ self.createReferenceSection(page, index)
return
else:
break
- # TODO: Make up a clever way of handling this.
- wikipedia.output(u'Found no section that can be preceeded by a new references section. Please add a references section manually.')
+ # This gets complicated: we want to place the new references
+ # section over the interwiki links and categories, but also
+ # over all navigation bars, persondata, and other templates
+ # that are at the bottom of the page. So we need some advanced
+ # regex magic.
+ # The strategy is: create a temporary copy of the text. From that,
+ # keep removing interwiki links, templates etc. from the bottom.
+ # At the end, look at the length of the temp text. That's the position
+ # where we'll insert the references section.
+ catNamespaces = '|'.join(page.site().category_namespaces())
+ categoryPattern = r'\[\[\s*(%s)\s*:[^\n]*\]\]\s*' % catNamespaces
+ interwikiPattern = r'\[\[([a-zA-Z\-]+)\s?:([^\[\]\n]*)\]\]\s*'
+ # won't work with nested templates
+ templatePattern = r'{{((?!}}).)+?}}\s*' # the negative lookahead assures that we'll match the last template occurence in the temp text.
+ commentPattern = r'<!--((?!-->).)*?-->\s*'
+ metadataR = re.compile(r'(\r\n)?(%s|%s|%s|%s)$' % (categoryPattern, interwikiPattern, templatePattern, commentPattern), re.DOTALL)
+ tmpText = oldText
+ while True:
+ match = metadataR.search(tmpText)
+ if match:
+ tmpText = tmpText[:match.start()]
+ else:
+ break
+ wikipedia.output(u'Found no section that can be preceeded by a new references section. Placing it before interwiki links, categories, and bottom templates.')
+ index = len(tmpText)
+ self.createReferenceSection(page, index)
+ def createReferenceSection(self, page, index):
+ oldText = page.get()
+ newSection = u'\n== %s ==\n\n<references/>\n' % wikipedia.translate(page.site(), referencesSections)[0]
+ newText = oldText[:index] + newSection + oldText[index:]
+ self.save(page, newText)
+
def save(self, page, newText):
"""
Saves the page to the wiki, if the user accepts the changes made.
Patches item #1792406, was opened at 2007-09-11 18:01
Message generated for change (Settings changed) made by ebola_rulez
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1792406&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Aurimas Fischer (ebola_rulez)
>Assigned to: Daniel Herding (wikipedian)
Summary: noreferences.py LT translation
Initial Comment:
Translation for lt.wiki
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1792406&group_…
Patches item #1792406, was opened at 2007-09-11 18:01
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1792406&group_…
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Aurimas Fischer (ebola_rulez)
Assigned to: Nobody/Anonymous (nobody)
Summary: noreferences.py LT translation
Initial Comment:
Translation for lt.wiki
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1792406&group_…
Revision: 4236
Author: wikipedian
Date: 2007-09-11 13:46:05 +0000 (Tue, 11 Sep 2007)
Log Message:
-----------
docu
Modified Paths:
--------------
trunk/pywikipedia/noreferences.py
Modified: trunk/pywikipedia/noreferences.py
===================================================================
--- trunk/pywikipedia/noreferences.py 2007-09-11 12:57:06 UTC (rev 4235)
+++ trunk/pywikipedia/noreferences.py 2007-09-11 13:46:05 UTC (rev 4236)
@@ -53,7 +53,8 @@
# link sections. This dictionary defines these sections, sorted by priority.
# For example, on an English wiki, the script would place the "References"
# section in front of the "Further reading" section, if that existed.
-# Otherwise, it would try to do it
+# Otherwise, it would try to put it in front of the "External links" section,
+# or if that fails, the "See also" section, etc.
placeBeforeSections = {
'de': [ # no explicit policy on where to put the references
u'Literatur',
@@ -68,7 +69,9 @@
],
}
-# How the references section should look like.
+# Titles of sections where a reference tag would fit into.
+# The first title should be the preferred one: It's the one that
+# will be used when a new section has to be created.
referencesSections = {
'de': [
u'Einzelnachweise', # The "Einzelnachweise" title is disputed, some people prefer the other variants
@@ -204,8 +207,8 @@
return
else:
break
- # TODO: Think of a clever way of handling this.
- wikipedia.output(u'Found no section that can be preceeded by a new references section. Please add a references section.')
+ # TODO: Make up a clever way of handling this.
+ wikipedia.output(u'Found no section that can be preceeded by a new references section. Please add a references section manually.')
def save(self, page, newText):
"""