Pywikipedia-l November 2008

pywikipedia-l@lists.wikimedia.org

20 participants
178 discussions

SVN: [6089] trunk/pywikipedia/djvutext.py
by nicdumz＠svn.wikimedia.org 12 Nov '08

12 Nov '08

Revision: 6089 Author: nicdumz Date: 2008-11-12 17:39:18 +0000 (Wed, 12 Nov 2008) Log Message: ----------- fixing outputs : Unicode conversions + missing parameter Modified Paths: -------------- trunk/pywikipedia/djvutext.py Modified: trunk/pywikipedia/djvutext.py =================================================================== --- trunk/pywikipedia/djvutext.py 2008-11-12 11:19:15 UTC (rev 6088) +++ trunk/pywikipedia/djvutext.py 2008-11-12 17:39:18 UTC (rev 6089) @@ -222,7 +222,7 @@ index_page = wikipedia.Page(site, index) if site.family.name != 'wikisource': - raise wikipedia.PageNotFound("Found family '%s'; Wikisource required.") + raise wikipedia.PageNotFound(u"Found family '%s'; Wikisource required." % site.family.name) if not index_page.exists() and index_page.namespace() == 0: index_namespace = wikipedia.Page(site, 'MediaWiki:Proofreadpage index namespace').get() @@ -233,7 +233,7 @@ if not index_page.exists(): raise wikipedia.NoPage("Page '%s' does not exist" % index) - wikipedia.output("uploading text from %s to %s" % (djvu, index_page) ) + wikipedia.output(u"uploading text from %s to %s" % (djvu, index_page) ) bot = DjVuTextBot(djvu, index, pages) if not bot.has_text():

1 0

[ pywikipediabot-Bugs-2269688 ] Unicode error with djvutext.py
by SourceForge.net 12 Nov '08

12 Nov '08

Bugs item #2269688, was opened at 2008-11-12 15:03 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2269688&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Forget (yannforget) Assigned to: Nobody/Anonymous (nobody) Summary: Unicode error with djvutext.py Initial Comment: on fr.wikisource: python djvutext.py -index:Livre:Le_Th%C3%A9%C3%A2tre_de_la_R%C3%A9volution._Le_Quatorze_Juillet._Danton._Les_Loups.djvu -djvu:Le_quatorze_juillet_Danton_Les_loups.djvu -pages:375 Checked for running processes. 1 processes currently running, including the current process. Traceback (most recent call last): File "djvutext.py", line 249, in <module> main() File "djvutext.py", line 236, in main wikipedia.output("uploading text from %s to %s" % (djvu, index_page) ) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 13: ordinal not in range(128) python version.py Pywikipedia [http] trunk/pywikipedia (r6084, Nov 11 2008, 21:51:31) Python 2.5.2 (r252, Sep 13 2008, 22:55:01) [GCC 4.1.2 (Gentoo 4.1.2 p1.1)] ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2269688&group_…

1 0

[ pywikipediabot-Feature Requests-2269013 ] Add options -cat and -file to fixing_redirects.py
by SourceForge.net 12 Nov '08

12 Nov '08

Feature Requests item #2269013, was opened at 2008-11-12 12:42 Message generated for change (Comment added) made by yannforget You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=2269013&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Priority: 5 Private: No Submitted By: Yann Forget (yannforget) Assigned to: Nobody/Anonymous (nobody) Summary: Add options -cat and -file to fixing_redirects.py Initial Comment: Hello, Please add the options -cat (changing all pages in a category) and -file (changing all pages listed in a file) to fixing_redirects.py Thanks, Yann ---------------------------------------------------------------------- >Comment By: Yann Forget (yannforget) Date: 2008-11-12 12:44 Message: and also to movepages.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=2269013&group_…

1 0

[ pywikipediabot-Feature Requests-2269013 ] Add options -cat and -file to fixing_redirects.py
by SourceForge.net 12 Nov '08

12 Nov '08

Feature Requests item #2269013, was opened at 2008-11-12 12:42 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=2269013&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Priority: 5 Private: No Submitted By: Yann Forget (yannforget) Assigned to: Nobody/Anonymous (nobody) Summary: Add options -cat and -file to fixing_redirects.py Initial Comment: Hello, Please add the options -cat (changing all pages in a category) and -file (changing all pages listed in a file) to fixing_redirects.py Thanks, Yann ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=2269013&group_…

1 0

SVN: [6088] trunk/pywikipedia/imagerecat.py
by multichill＠svn.wikimedia.org 12 Nov '08

12 Nov '08

Revision: 6088 Author: multichill Date: 2008-11-12 11:19:15 +0000 (Wed, 12 Nov 2008) Log Message: ----------- Catch IOError when filtering parents Modified Paths: -------------- trunk/pywikipedia/imagerecat.py Modified: trunk/pywikipedia/imagerecat.py =================================================================== --- trunk/pywikipedia/imagerecat.py 2008-11-12 11:10:52 UTC (rev 6087) +++ trunk/pywikipedia/imagerecat.py 2008-11-12 11:19:15 UTC (rev 6088) @@ -234,11 +234,14 @@ toFilter = toFilter + "[[Category:" + cat + "]]\n" #try: parameters = urllib.urlencode({'source' : toFilter.encode('utf-8'), 'bot' : '1'}) - filterCategoriesPage = urllib.urlopen("http://toolserver.org/~multichill/filtercats.php?%s" % parameters) - #print filterCategoriesPage.read().decode('utf-8') filterCategoriesRe = re.compile('\[\[Category:([^\]]*)\]\]') - result = filterCategoriesRe.findall(filterCategoriesPage.read().decode('utf-8')) - #except: + try: + filterCategoriesPage = urllib.urlopen("http://toolserver.org/~multichill/filtercats.php?%s" % parameters) + result = filterCategoriesRe.findall(filterCategoriesPage.read().decode('utf-8')) + except IOError: + #Something is wrong, forget about this filter and just return the input + return categories + if not result: #Is empty, dont want to remove all categories return categories

1 0

SVN: [6087] trunk/pywikipedia/imagerecat.py
by multichill＠svn.wikimedia.org 12 Nov '08

12 Nov '08

Revision: 6087 Author: multichill Date: 2008-11-12 11:10:52 +0000 (Wed, 12 Nov 2008) Log Message: ----------- cat = cat.replace('_', ' ') Modified Paths: -------------- trunk/pywikipedia/imagerecat.py Modified: trunk/pywikipedia/imagerecat.py =================================================================== --- trunk/pywikipedia/imagerecat.py 2008-11-11 22:13:28 UTC (rev 6086) +++ trunk/pywikipedia/imagerecat.py 2008-11-12 11:10:52 UTC (rev 6087) @@ -158,6 +158,7 @@ ''' result = [] for cat in categories: + cat = cat.replace('_', ' ') if (cat not in category_blacklist): result.append(cat) return list(set(result))

1 0

[ pywikipediabot-Bugs-2193942 ] reading category: memory leak and slow down
by SourceForge.net 12 Nov '08

12 Nov '08

Bugs item #2193942, was opened at 2008-10-25 11:10 Message generated for change (Comment added) made by sf-robot You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2193942&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: category Group: None >Status: Closed Resolution: Fixed Priority: 5 Private: No Submitted By: Simone Malacarne (smalacarne) Assigned to: Nobody/Anonymous (nobody) Summary: reading category: memory leak and slow down Initial Comment: I need to read a very big category (80.000+ articles). So i just do: site = wikipedia.getSite() cat = catlib.Category(site,'category name') gen = pagegenerators.PreloadingGenerator(cat.articles(), pageNumber=100) for page in gen: do_something problem is that the program start using more and more memory (at the end near 2giga ram). Even cpu time increase over time, if first 10.000 articles are processed in 10 min, second 10.000 double that time and so on... it takes about 20 hours to read all the articles. If i use: gen = pagegenerators.CategorizedPageGenerator(cat , recurse=False, start=u'') instead of PreloadingGenerator i dont have mem or cpu leaks but it's slow as hell to read and articles at the time (more than 24 hours to finish). Pywikipedia [http] trunk/pywikipedia (r6015, Oct 24 2008, 18:29:39) Python 2.5.2 (r252:60911, Oct 5 2008, 19:29:17) [GCC 4.3.2] ---------------------------------------------------------------------- >Comment By: SourceForge Robot (sf-robot) Date: 2008-11-12 02:20 Message: This Tracker item was closed automatically by the system. It was previously set to a Pending status, and the original submitter did not respond within 14 days (the time period specified by the administrator of this Tracker). ---------------------------------------------------------------------- Comment By: NicDumZ Nicolas Dumazet (nicdumz) Date: 2008-10-28 10:30 Message: Well, guess what ? I have no idea why we would need to cache the content of a category... I guess someone assumed users would iterate through a category several times. Does anyone has a serious usage case of such a behavior ? I might be wrong but I think that you can always serialize in some way your code to avoid calling several times your generator function. Since r6038, the default generator now uses a naive content getter which does not cache anything. ---------------------------------------------------------------------- Comment By: Simone Malacarne (smalacarne) Date: 2008-10-26 20:43 Message: I track the problem to catlib in the category._getContents function. The function cache something but with a lot of pages memory and cpu use is massive. I try to comment 2 lines in this part: else: print ('not Cached') for tag, page in self._parseCategory(purge, startFrom): if tag == ARTICLE: #self.articleCache.append(page) if not page in cache: #cache.append(page) yield ARTICLE, page and all is fine now, memory use is about 20/30mbyte fix and cpu occupation is normal. Don't know what that cache is used for but it caused me lot of trouble. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=2193942&group_…

1 0

SVN: [6086] trunk/pywikipedia
by erwin85＠svn.wikimedia.org 12 Nov '08

12 Nov '08

Revision: 6086 Author: erwin85 Date: 2008-11-11 22:13:28 +0000 (Tue, 11 Nov 2008) Log Message: ----------- Undoing r6085. MediaWiki will complain that you're recreating the page. Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Property Changed: ---------------- trunk/pywikipedia/commons_category_redirect.py Property changes on: trunk/pywikipedia/commons_category_redirect.py ___________________________________________________________________ Deleted: svn:mergeinfo - Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-11-11 21:55:29 UTC (rev 6085) +++ trunk/pywikipedia/wikipedia.py 2008-11-11 22:13:28 UTC (rev 6086) @@ -461,8 +461,7 @@ self._permalink = None self._userName = None self._ipedit = None - self._editTime = '0' - self._startTime = '0' + self._editTime = None # For the Flagged Revisions MediaWiki extension self._revisionId = None self._deletedRevs = None

1 0

SVN: [6085] trunk/pywikipedia/wikipedia.py
by erwin85＠svn.wikimedia.org 11 Nov '08

11 Nov '08

Revision: 6085 Author: erwin85 Date: 2008-11-11 21:55:29 +0000 (Tue, 11 Nov 2008) Log Message: ----------- Fixed errors about unset edit and start time for non-existing pages. Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-11-11 21:51:31 UTC (rev 6084) +++ trunk/pywikipedia/wikipedia.py 2008-11-11 21:55:29 UTC (rev 6085) @@ -461,7 +461,8 @@ self._permalink = None self._userName = None self._ipedit = None - self._editTime = None + self._editTime = '0' + self._startTime = '0' # For the Flagged Revisions MediaWiki extension self._revisionId = None self._deletedRevs = None

1 0

SVN: [6084] trunk/pywikipedia/wikipedia.py
by multichill＠svn.wikimedia.org 11 Nov '08

11 Nov '08

Revision: 6084 Author: multichill Date: 2008-11-11 21:51:31 +0000 (Tue, 11 Nov 2008) Log Message: ----------- Adding leaveRedirect to move (default True). Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-11-11 21:42:04 UTC (rev 6083) +++ trunk/pywikipedia/wikipedia.py 2008-11-11 21:51:31 UTC (rev 6084) @@ -2183,7 +2183,7 @@ return users def move(self, newtitle, reason=None, movetalkpage=True, sysop=False, - throttle=True, deleteAndMove=False, safe=True, fixredirects=True): + throttle=True, deleteAndMove=False, safe=True, fixredirects=True, leaveRedirect=True): """Move this page to new title given by newtitle. If safe, don't try to move and delete if not directly requested. @@ -2226,6 +2226,10 @@ predata['wpFixRedirects'] = '1' else: predata['wpFixRedirects'] = '0' + if leaveRedirect: + predata['wpLeaveRedirect'] = '1' + else: + predata['wpLeaveRedirect'] = '0' if token: predata['wpEditToken'] = token if self.site().hostname() in config.authenticate.keys():

1 0

← Newer
1
...
9
10
11
12
13
14
15
...
18
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Pywikipedia-l November 2008