pywikibot

pywikibot@lists.wikimedia.org

3 participants
6830 discussions

[Pywikipedia-l] SVN: [4659] trunk/pywikipedia/wikipedia.py
by rotem＠svn.wikimedia.org 09 Dec '07

09 Dec '07

Revision: 4659 Author: rotem Date: 2007-12-09 13:58:26 +0000 (Sun, 09 Dec 2007) Log Message: ----------- (bug 1844339) Avoid AttributeError in sites which use authentication. Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2007-12-09 13:54:30 UTC (rev 4658) +++ trunk/pywikipedia/wikipedia.py 2007-12-09 13:58:26 UTC (rev 4659) @@ -1244,8 +1244,8 @@ output(u'Changing page %s' % self.aslink(forceInterwiki=True)) # Submit the prepared information if self.site().hostname() in config.authenticate.keys(): - predata.append(("Content-type","application/x-www-form-urlencoded")) - predata.append(("User-agent", useragent)) + predata["Content-type"] = "application/x-www-form-urlencoded" + predata["User-agent"] = useragent data = self.site().urlEncode(predata) response = urllib2.urlopen(urllib2.Request(self.site().protocol() + '://' + self.site().hostname() + address, data)) # I'm not sure what to check in this case, so I just assume

1 0

[Pywikipedia-l] [ pywikipediabot-Feature Requests-1844225 ] Clean Sandbox's talkpage
by SourceForge.net 09 Dec '07

09 Dec '07

Feature Requests item #1844225, was opened at 2007-12-04 19:17 Message generated for change (Comment added) made by rotemliss You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1844225&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Priority: 5 Private: No Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: Clean Sandbox's talkpage Initial Comment: Hello! Some wikis (like English Wikipedia, Arabic Wikipedia etc..) use Sandbox's talkpage (or other pages) for sandboxing. I think we can't add more than one sandbox in clean_sandbox.py. can someone fix this problem? ---------------------------------------------------------------------- Comment By: Rotem Liss (rotemliss) Date: 2007-12-09 15:55 Message: Logged In: YES user_id=1327030 Originator: NO Fixed in r4658. They can now be added as a list: [u'Page', u'Page2'] ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1844225&group_…

1 0

[Pywikipedia-l] SVN: [4658] trunk/pywikipedia/clean_sandbox.py
by rotem＠svn.wikimedia.org 09 Dec '07

09 Dec '07

Revision: 4658 Author: rotem Date: 2007-12-09 13:54:30 +0000 (Sun, 09 Dec 2007) Log Message: ----------- (feature request 1844225) Make it possible to add a list of pages to sandboxTitle. Modified Paths: -------------- trunk/pywikipedia/clean_sandbox.py Modified: trunk/pywikipedia/clean_sandbox.py =================================================================== --- trunk/pywikipedia/clean_sandbox.py 2007-12-09 13:46:39 UTC (rev 4657) +++ trunk/pywikipedia/clean_sandbox.py 2007-12-09 13:54:30 UTC (rev 4658) @@ -28,7 +28,7 @@ 'ar': u'{{من فضلك اترك هذا السطر ولا تعدله (عنوان ساحة اللعب)}}\n', 'de': u'{{Bitte erst NACH dieser Zeile schreiben! (Begrüßungskasten)}}\r\n', 'en': u'{{Please leave this line alone (sandbox heading)}}\n ', - 'he': u'{{ארגז חול}}\n', + 'he': u'{{ארגז חול}}\n', 'it': u'{{sandbox}} ', 'ja': u'{{subst:サンドボックス}}', 'ko': u'{{연습장 안내문}}', @@ -81,17 +81,22 @@ while True: now = time.strftime("%d %b %Y %H:%M:%S (UTC)", time.gmtime()) localSandboxTitle = wikipedia.translate(mySite, sandboxTitle) - sandboxPage = wikipedia.Page(mySite, localSandboxTitle) - try: - text = sandboxPage.get() - translatedContent = wikipedia.translate(mySite, content) - if text.strip() == translatedContent.strip(): - wikipedia.output(u'The sandbox is still clean, no change necessary.') - else: - translatedMsg = wikipedia.translate(mySite, msg) - sandboxPage.put(translatedContent, translatedMsg) - except wikipedia.EditConflict: - wikipedia.output(u'*** Loading again because of edit conflict.\n') + if type(localSandboxTitle) is list: + titles = localSandboxTitle + else: + titles = [localSandboxTitle,] + for title in titles: + sandboxPage = wikipedia.Page(mySite, localSandboxTitle) + try: + text = sandboxPage.get() + translatedContent = wikipedia.translate(mySite, content) + if text.strip() == translatedContent.strip(): + wikipedia.output(u'The sandbox is still clean, no change necessary.') + else: + translatedMsg = wikipedia.translate(mySite, msg) + sandboxPage.put(translatedContent, translatedMsg) + except wikipedia.EditConflict: + wikipedia.output(u'*** Loading again because of edit conflict.\n') if self.no_repeat: wikipedia.output(u'\nDone.') wikipedia.stopme()

1 0

[Pywikipedia-l] SVN: [4657] trunk/pywikipedia/family.py
by rotem＠svn.wikimedia.org 09 Dec '07

09 Dec '07

Revision: 4657 Author: rotem Date: 2007-12-09 13:46:39 +0000 (Sun, 09 Dec 2007) Log Message: ----------- Cleaner code. Modified Paths: -------------- trunk/pywikipedia/family.py Modified: trunk/pywikipedia/family.py =================================================================== --- trunk/pywikipedia/family.py 2007-12-09 13:28:18 UTC (rev 4656) +++ trunk/pywikipedia/family.py 2007-12-09 13:46:39 UTC (rev 4657) @@ -2403,12 +2403,12 @@ raise KeyError('ERROR: title for namespace %d in language %s unknown' % (ns_number, code)) if all: - if type(v) == type([]): + if type(v) is list: return tuple(v) else: return (v, ) else: - if type(v) == type([]): + if type(v) is list: return v[0] else: return v

1 0

[Pywikipedia-l] [ pywikipediabot-Patches-1843789 ] making image.py work
by SourceForge.net 09 Dec '07

09 Dec '07

Patches item #1843789, was opened at 2007-12-04 04:20 Message generated for change (Comment added) made by rotemliss You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1843789&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 7 Private: No Submitted By: Pietro Battiston (toobaz) Assigned to: Nobody/Anonymous (nobody) Summary: making image.py work Initial Comment: The following command: python image.py pippo.png gives the following output: Checked for running processes. 1 processes currently running, including the current process. 'Page' object has no attribute 'usingPages' The problem is that oldImagePage is an instance of wikipedia.Page instead of wikipedia.ImagePage. I attach a very simple patch that fixes it. Then, everything works. ---------------------------------------------------------------------- Comment By: Rotem Liss (rotemliss) Date: 2007-12-09 15:39 Message: Logged In: YES user_id=1327030 Originator: NO This is duplicate to bug 1843759. I fixed it. ---------------------------------------------------------------------- Comment By: Pietro Battiston (toobaz) Date: 2007-12-08 13:37 Message: Logged In: YES user_id=1553460 Originator: YES Sorry, the original summary was totally wrong. The problem is in image.py, not replace.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1843789&group_…

1 0

[Pywikipedia-l] [ pywikipediabot-Patches-1843787 ] catlib _getContentsAndSupercats performance issue
by SourceForge.net 09 Dec '07

09 Dec '07

Patches item #1843787, was opened at 2007-12-04 04:18 Message generated for change (Comment added) made by rotemliss You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1843787&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Pietro Battiston (toobaz) Assigned to: Nobody/Anonymous (nobody) Summary: catlib _getContentsAndSupercats performance issue Initial Comment: catlib.py's _getContentsAndSupercats method has a performance issue that in some cases can slow a lot the process of recursiverly downloading all pages or subcategories of a category. See this example (chosen just because it's short to report, not because it's so pathological): ###########ipython output############### In [1]: import catlib Checked for running processes. 1 processes currently running, including the current process. In [2]: len(catlib.Category('it', 'Categoria:Geometria descrittiva').articlesList(recurse=True)) Getting [[Categoria:Geometria descrittiva]]... Getting [[Categoria:Coperture a volta]]... Getting [[Categoria:Corrispondenza biunivoca (geometria descrittiva)]]... Getting [[Categoria:Curve piane]]... Getting [[Categoria:Curve tridimensionali]]... Getting [[Categoria:Glossario (geometria descrittiva)]]... Getting [[Categoria:Metodi di rappresentazione]]... Getting [[Categoria:Modellazione geometrica]]... Getting [[Categoria:Tassellazioni]]... Getting [[Categoria:Poliedri]]... Getting [[Categoria:Tassellazioni]]... Getting [[Categoria:Problemi di misura]]... Getting [[Categoria:Stub geometria descrittiva]]... Getting [[Categoria:Superfici]]... Getting [[Categoria:Sviluppo di solidi]]... Getting [[Categoria:Tangenza]]... Out[2]: 393 ###########end ipython output############### As you can see, [[Categoria:Tassellazioni]] is downloaded 2 times. But I can grant you that there are a lot of much worse cases. Anyway, I'm attaching a patch. After the patch, here are the same commands: ###########ipython output############### In [1]: import catlib Checked for running processes. 1 processes currently running, including the current process. In [2]: len(catlib.Category('it', 'Categoria:Geometria descrittiva').articlesList(recurse=True)) Getting [[Categoria:Geometria descrittiva]]... Getting [[Categoria:Coperture a volta]]... Getting [[Categoria:Corrispondenza biunivoca (geometria descrittiva)]]... Getting [[Categoria:Curve piane]]... Getting [[Categoria:Curve tridimensionali]]... Getting [[Categoria:Glossario (geometria descrittiva)]]... Getting [[Categoria:Metodi di rappresentazione]]... Getting [[Categoria:Modellazione geometrica]]... Getting [[Categoria:Tassellazioni]]... Getting [[Categoria:Poliedri]]... Getting [[Categoria:Problemi di misura]]... Getting [[Categoria:Stub geometria descrittiva]]... Getting [[Categoria:Superfici]]... Getting [[Categoria:Sviluppo di solidi]]... Getting [[Categoria:Tangenza]]... Out[2]: 393 ###########end ipython output############### Notice this patch also solves the problem of eventual loops in categories: catlib won't loop. ---------------------------------------------------------------------- Comment By: Rotem Liss (rotemliss) Date: 2007-12-09 15:37 Message: Logged In: YES user_id=1327030 Originator: NO I changed the patch so that it solves also the problem of duplicate returns, and also does add the subcategory to its cache, and applied it in r4656. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1843787&group_…

1 0

[Pywikipedia-l] SVN: [4656] trunk/pywikipedia/catlib.py
by rotem＠svn.wikimedia.org 09 Dec '07

09 Dec '07

Revision: 4656 Author: rotem Date: 2007-12-09 13:28:18 +0000 (Sun, 09 Dec 2007) Log Message: ----------- (patch 1843787) catlib _getContentsAndSupercats performance issue (changed patch to avoid possible problems in caching) Modified Paths: -------------- trunk/pywikipedia/catlib.py Modified: trunk/pywikipedia/catlib.py =================================================================== --- trunk/pywikipedia/catlib.py 2007-12-09 13:25:52 UTC (rev 4655) +++ trunk/pywikipedia/catlib.py 2007-12-09 13:28:18 UTC (rev 4656) @@ -83,7 +83,7 @@ return '[[%s]]' % titleWithSortKey def _getContentsAndSupercats(self, recurse=False, purge=False, - startFrom=None): + startFrom=None, cache=[]): """ Cache results of _parseCategory for a second call. @@ -107,36 +107,46 @@ newrecurse = recurse if self.completelyCached: for article in self.articleCache: - yield ARTICLE, article + if article not in cache: + cache.append(article) + yield ARTICLE, article for subcat in self.subcatCache: - yield SUBCATEGORY, subcat - if recurse: - # contents of subcategory are cached by calling - # this method recursively; therefore, do not cache - # them again - for item in subcat._getContentsAndSupercats(newrecurse, + if subcat not in cache: + cache.append(subcat) + yield SUBCATEGORY, subcat + if recurse: + # contents of subcategory are cached by calling + # this method recursively; therefore, do not cache + # them again + for item in subcat._getContentsAndSupercats(newrecurse, purge): - if item[0] != SUPERCATEGORY: - yield item + if item[0] != SUPERCATEGORY: + yield item for supercat in self.supercatCache: yield SUPERCATEGORY, supercat else: for tag, page in self._parseCategory(purge, startFrom): if tag == ARTICLE: self.articleCache.append(page) + if not page in cache: + cache.append(page) + yield ARTICLE, page elif tag == SUBCATEGORY: self.subcatCache.append(page) - if recurse: - # contents of subcategory are cached by calling - # this method recursively; therefore, do not cache - # them again - for item in page._getContentsAndSupercats(newrecurse, + if not page in cache: + cache.append(page) + yield SUBCATEGORY, page + if recurse: + # contents of subcategory are cached by calling + # this method recursively; therefore, do not cache + # them again + for item in page._getContentsAndSupercats(newrecurse, purge): - if item[0] != SUPERCATEGORY: - yield item + if item[0] != SUPERCATEGORY: + yield item elif tag == SUPERCATEGORY: self.supercatCache.append(page) - yield tag, page + yield SUPERCATEGORY, page if not startFrom: self.completelyCached = True

1 0

[Pywikipedia-l] SVN: [4655] trunk/pywikipedia/wikipedia.py
by filnik＠svn.wikimedia.org 09 Dec '07

09 Dec '07

Revision: 4655 Author: filnik Date: 2007-12-09 13:25:52 +0000 (Sun, 09 Dec 2007) Log Message: ----------- Bugfix (if repeat is not used it returns only one image) Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2007-12-09 13:23:16 UTC (rev 4654) +++ trunk/pywikipedia/wikipedia.py 2007-12-09 13:25:52 UTC (rev 4655) @@ -4227,21 +4227,22 @@ re.UNICODE) pos = 0 seen = list() - ext_list = list() - for m in regexp.finditer(html): - new = m.group('new') - im = m.group('image') - ext = m.group('ext') - # This prevent pages with strange characters. They will be loaded without problem. - image = "%s.%s" % (im, ext) - if new != '': - output(u"Skipping %s because it has been deleted." % image) + ext_list = list() + while True: + for m in regexp.finditer(html): + new = m.group('new') + im = m.group('image') + ext = m.group('ext') + # This prevent pages with strange characters. They will be loaded without problem. + image = "%s.%s" % (im, ext) + if new != '': + output(u"Skipping %s because it has been deleted." % image) + if image not in seen: + seen.append(image) if image not in seen: seen.append(image) - if image not in seen: - seen.append(image) - page = Page(self, 'Image:%s' % image) - yield page + page = Page(self, 'Image:%s' % image) + yield page if not repeat: output(u"\t\t>> All images checked. <<") break

1 0

[Pywikipedia-l] SVN: [4654] trunk/pywikipedia/pagegenerators.py
by filnik＠svn.wikimedia.org 09 Dec '07

09 Dec '07

Revision: 4654 Author: filnik Date: 2007-12-09 13:23:16 +0000 (Sun, 09 Dec 2007) Log Message: ----------- Adding newImages() in the GenFactory class Modified Paths: -------------- trunk/pywikipedia/pagegenerators.py Modified: trunk/pywikipedia/pagegenerators.py =================================================================== --- trunk/pywikipedia/pagegenerators.py 2007-12-09 13:04:42 UTC (rev 4653) +++ trunk/pywikipedia/pagegenerators.py 2007-12-09 13:23:16 UTC (rev 4654) @@ -772,6 +772,12 @@ if namespace: prefix = prefix[colon+1:] gen = PrefixingPageGenerator(prefix = prefix, namespace = namespace) + elif arg.startswith('-newimages'): + if len(arg) == 10: + limit = wikipedia.input(u'How many images do you want to check?') + else: + limit = arg[11:] + gen = newImages(limit, wikipedia.getSite()) elif arg.startswith('-new'): if len(arg) >=5: gen = NewpagesPageGenerator(number = int(arg[5:]))

1 0

[Pywikipedia-l] SVN: [4653] trunk/pywikipedia/wikipedia.py
by filnik＠svn.wikimedia.org 09 Dec '07

09 Dec '07

Revision: 4653 Author: filnik Date: 2007-12-09 13:04:42 +0000 (Sun, 09 Dec 2007) Log Message: ----------- Bugfix (wikipedia.output() is only output() in the wikipedia.py file...) Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2007-12-09 12:28:55 UTC (rev 4652) +++ trunk/pywikipedia/wikipedia.py 2007-12-09 13:04:42 UTC (rev 4653) @@ -4235,7 +4235,7 @@ # This prevent pages with strange characters. They will be loaded without problem. image = "%s.%s" % (im, ext) if new != '': - wikipedia.output(u"Skipping %s because it has been deleted." % image) + output(u"Skipping %s because it has been deleted." % image) if image not in seen: seen.append(image) if image not in seen: @@ -4243,7 +4243,7 @@ page = Page(self, 'Image:%s' % image) yield page if not repeat: - wikipedia.output(u"\t\t>> All images checked. <<") + output(u"\t\t>> All images checked. <<") break def uncategorizedimages(self, number = 10, repeat = False):

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot