pywikibot

pywikibot@lists.wikimedia.org

3 participants
6830 discussions

Re: [Pywikipedia-l] SVN: [4507] branches/rewrite/pywikibot/data/api.py
by Merlijn van Deen 06 Nov '07

06 Nov '07

> Revision: 4507 > Author: russblau > Date: 2007-11-05 15:13:10 +0000 (Mon, 05 Nov 2007) > > Log Message: > ----------- > Generalize query method; add some response processing and error handling. Although I appreciate your efforts, I still want to ask you to wait with any other rewrite commits until you have read, understood, and reacted to this (fairly long) email. The purpose of the rewrite was a) to restructure the framework b) to have consistent formatting and documentation c) to move to the API and possibly d) to add i18n support First of all: point b. Before any new code is written, we should agree on a fixed coding style. The current framework has several coding styles used throughout the framework - and even sometimes in one class: in the Page class, we have got Page.urlname, Page.isAutoTitle and Page.put_async... - I think we should use one style for the entire framework. For the coding style, I propose the following: * As base: PEP 8 [1]. * UTF8 encoding for all files (and using u'' for all strings), with UTF8 BOM * Docstrings mandatory, using Epydoc [2] for describing parameters and return value. Defining the function in this way almost automatically defines unit tests for that function. * __version__ in every file, and setting the corresponding SVN property. * One module per class Note that all code should be written thread safe: we want to use persistent HTTP connections and this means we will be sharing a connection throughout several page objects. If we want to be able to put in a thread, we really need to make sure code is thread safe. Secondly: point a. The point of the rewrite was restructuring the framework. Adding structure after writing is much harder than first thinking of structure and then coding along the structure. First of all, we need to split the framework and the bots. Using the name 'pywikibot' for the framework might not be the best idea because of that ;) Because adding structure afterwards is much harder, I think we first should decide on what modules/classes we want, then defining what functions we want and what these functions should do. After that, we can start writing unit tests and functions. Unit tests are there to help defining what code is necessary. In general, first define what a function should to, then write unit tests, then write code to make these tests work, and stop coding when all tests pass. If a function needs to do more, first update the defenition, then update the unit tests, then update the code. This makes sure a) that documentation is always up to date with the code and b) that a change does not break existing behavior. For more information about unit testing and how to code using them, please read chapter 13, 14 and 15 of Dive Into Python [3]. For unit testing, we will need one or more installs of MediaWiki; the SVN release used at wikipedia, but maybe also older stable versions - if we want to maintain compatibility. On a compatibility note; even though we are changing the way the framework works, it should be possible to create a layer that maps functions in the old framework to the new framework. thirdly: point c. This really is not a seperate point, but it is important. I think we should use the API where possible, but having a fallback to the monobook html parser. The API allows neat ways of getting information from many pages at once. We will need a page generator that can handle this information, and that will generate page objects according to the information recieved. This will mean we need more data on what information we have got already etc. finally: point d. This is not a very important point, but it's kinda interesting. With the new framework, it's easier to restructure existing functions, making translations easier. I'm sorry for the long email, and I repeat: I really appreciate your efforts, but I really think we should address these issues before starting at - even the most basic - code. --valhallasw [1] http://www.python.org/dev/peps/pep-0008/ [2] http://epydoc.sourceforge.net/manual-epytext.html [3] http://diveintopython.org/unit_testing/index.html

3 2

Re: [Pywikipedia-l] Rewrite thoughts
by Misza13 06 Nov '07

06 Nov '07

Ok now. As I am a complete lamer when it comes to team coding, coding standards etc., I'll take your words for it, but have nonetheless several lame questions. The purpose of the rewrite was > a) to restructure the framework > b) to have consistent formatting and documentation > c) to move to the API > and possibly > d) to add i18n support > > First of all: point b. > Before any new code is written, we should agree on a fixed coding style. > The current framework has several coding styles used throughout the framework - and even sometimes in one class: in the Page class, we have > got Page.urlname, Page.isAutoTitle and Page.put_async... - I think we > should use one style for the entire framework. If team coding is a democracy then I vote for camelCaseWithFirstWordLowercase. It's a strong standard in Java (which is a close cousin to Python) and is frequent in Python too (take Twisted for an example). For the coding style, I propose the following: > * As base: PEP 8 [1]. Fine. I suggest the most common 1 indentation level = 4 spaces. * UTF8 encoding for all files (and using u'' for all strings), with UTF8 BOM I'll assume gvim does that for me. o_O * Docstrings mandatory, using Epydoc [2] for describing parameters and > return value. Defining the function in this way almost automatically > defines unit tests for that function. That's an interesting document - smells of javadoc too. :) One module per class Java style again? ;) Anyway, let's make sure I understand a "module" correctly, especially in the context of unit tests later on. A module corresponds to one .py file (like http.py or api.py from our example) and a folder of these (data/) is a package, not module? Despite of this: >>> import data >>> data <module 'data' from 'data/__init__.py'> Note that all code should be written thread safe: we want to use > persistent HTTP connections and this means we will be sharing a connection > throughout several page objects. If we want to be able to put in a thread, > we really need to make sure code is thread safe. If I may suggest, let's leave data.http.HTTP (or its equivalent) as an object encapsulating a connection and presenting wrappers on GET and POST methods, but in a non-thread-safe manner and instead resolve this at a higher level (Site, perhaps) by keeping a pool of connections or something similar (or maybe write a special HTTPPool handler between Site and HTTP). That's really a matter of deciding whether the framework handles multithreading or do we tell people that each thread should have its own instance of Site. Secondly: point a. > <snip> Because adding structure afterwards is much harder, I think we first > should decide on what modules/classes we want, then defining what > functions we want and what these functions should do. After that, we can > start writing unit tests and functions. Where do we start? Edit http://www.botwiki.sno.cc/wiki/Rewrite and add classes until someone says enough? On a compatibility note; even though we are changing the way the framework > works, it should be possible to create a layer that maps functions in the > old framework to the new framework. Yes, I would assume most methods of current high-level objects (Page, Site, generators) will not change their interface much. thirdly: point c. Yes, <3 API - I think we established that already. :> > finally: point d. > This is not a very important point, but it's kinda interesting. With the > new framework, it's easier to restructure existing functions, making > translations easier. Ok now, can you elaborate on this? I don't think I'm getting the point - we already same i18n - bots have localized edit summaries, the framework knows #REDIRECT and namespace names locales as well. Are you talking about some whole new level of i18n? I'm sorry for the long email, and I repeat: I really appreciate your > efforts, but I really think we should address these issues before starting > at - even the most basic - code. Well, it seems I'm a coder, not project designer and it tempts me to stand back, watch the fireworks and return when we actually need some code. But instead, I hope to gain some experience in team programming. Misza

1 0

[Pywikipedia-l] SVN: [4510] trunk/pywikipedia/
by warddr＠svn.wikimedia.org 06 Nov '07

06 Nov '07

Revision: 4510 Author: warddr Date: 2007-11-05 17:03:48 +0000 (Mon, 05 Nov 2007) Log Message: ----------- added category.dump.bz2 to ignore list Property Changed: ---------------- trunk/pywikipedia/ Property changes on: trunk/pywikipedia ___________________________________________________________________ Name: svn:ignore - *.pyc user-config.py autonomous_problem.dat sax_parse_bug.dat treelang.log *.dump warning*.log *.txt check_extern.log throttle.log login-data password-file + *.pyc user-config.py autonomous_problem.dat sax_parse_bug.dat treelang.log *.dump warning*.log *.txt check_extern.log throttle.log login-data password-file category.dump.bz2

1 0

[Pywikipedia-l] SVN: [4509] trunk/pywikipedia/families/wikibond_family.py
by warddr＠svn.wikimedia.org 05 Nov '07

05 Nov '07

Revision: 4509 Author: warddr Date: 2007-11-05 16:53:21 +0000 (Mon, 05 Nov 2007) Log Message: ----------- copy - paste Modified Paths: -------------- trunk/pywikipedia/families/wikibond_family.py Modified: trunk/pywikipedia/families/wikibond_family.py =================================================================== --- trunk/pywikipedia/families/wikibond_family.py 2007-11-05 16:39:03 UTC (rev 4508) +++ trunk/pywikipedia/families/wikibond_family.py 2007-11-05 16:53:21 UTC (rev 4509) @@ -16,7 +16,7 @@ 'nl': [u'WikiBond'], } self.namespaces[5] = { - 'botwiki': [u'Overleg WikiBond'], + 'nl': [u'Overleg WikiBond'], } def path(self, code):

1 0

[Pywikipedia-l] SVN: [4508] trunk/pywikipedia/families/wikibond_family.py
by warddr＠svn.wikimedia.org 05 Nov '07

05 Nov '07

Revision: 4508 Author: warddr Date: 2007-11-05 16:39:03 +0000 (Mon, 05 Nov 2007) Log Message: ----------- +namespacess Modified Paths: -------------- trunk/pywikipedia/families/wikibond_family.py Modified: trunk/pywikipedia/families/wikibond_family.py =================================================================== --- trunk/pywikipedia/families/wikibond_family.py 2007-11-05 15:13:10 UTC (rev 4507) +++ trunk/pywikipedia/families/wikibond_family.py 2007-11-05 16:39:03 UTC (rev 4508) @@ -3,16 +3,21 @@ import family # I added this becouse someone asked me to. The url op the wiki: nl.wikibond.org -#to do: add namespaces class Family(family.Family): - + def __init__(self): family.Family.__init__(self) self.name = 'wikibond' self.langs = { 'nl': 'nl.wikibond.org', } + self.namespaces[4] = { + 'nl': [u'WikiBond'], + } + self.namespaces[5] = { + 'botwiki': [u'Overleg WikiBond'], + } def path(self, code): return '/wikibond/index.php'

1 0

[Pywikipedia-l] SVN: [4507] branches/rewrite/pywikibot/data/api.py
by russblau＠svn.wikimedia.org 05 Nov '07

05 Nov '07

Revision: 4507 Author: russblau Date: 2007-11-05 15:13:10 +0000 (Mon, 05 Nov 2007) Log Message: ----------- Generalize query method; add some response processing and error handling. Modified Paths: -------------- branches/rewrite/pywikibot/data/api.py Modified: branches/rewrite/pywikibot/data/api.py =================================================================== --- branches/rewrite/pywikibot/data/api.py 2007-11-05 10:23:03 UTC (rev 4506) +++ branches/rewrite/pywikibot/data/api.py 2007-11-05 15:13:10 UTC (rev 4507) @@ -12,17 +12,78 @@ import urllib import http +import simplejson as json +import warnings +class APIError(Exception): + """The wiki site returned an error message.""" + def __init__(self, errordict): + """Save error dict returned by MW API.""" + self.errors = errordict + def __str__(self): + return "%(code)s: %(info)s" % self.errors + + class API: def __init__(self, site): self.site = site - def query(prop, **kwargs): - params = dict(kwargs) - params['action'] = 'query' + def request(self, **params): if not params.has_key('format'): #Most probably, we want the JSON format params['format'] = 'json' + return http.HTTP(None).POST('/w/api.php',params) #TODO: Use site's HTTP object instead - return http.HTTP(None).POST('/w/api.php',params) #TODO: Use site's HTTP object instead + def query(self, **params): + if not params.has_key('action'): + params['action'] = 'query' + return self.request(**params) + + def query_response(self, **params): + """Submit a query and parse the response, returning a dict or None.""" + if params.has_key('format') and params['format'] != 'json': + raise TypeError("Query format '%s' cannot be parsed." % params['format']) + while True: + httpcode, rawdata = self.query(**params) + if httpcode != 200: + raise APIError( + {'code': httpcode, + 'info': "HTTP error code received.", + 'data': rawdata}) + if rawdata.startswith(u"unknown_action"): + e = {'code': data[:14], 'info': data[16:]} + raise APIError(e) + try: + result = json.loads(rawdata) + except ValueError: + # if the result isn't valid JSON, there must be a server + # problem. Wait a few seconds and try again + # TODO: implement a throttle + warnings.warn( +"Non-JSON response received from server %s; the server may be down." + % self.site) + continue + if type(result) is dict: + if result.has_key("error"): + # raise error + raise APIError(result['error']) + if result.has_key("query"): + return result + raise APIError( + {'code': "Unknown API error", + 'info': "Response received with no 'query' key.", + 'data': result}) + if type(result) is list: + if result == []: + return None + raise APIError( + {'code': "Unknown API error", + 'info': "Query returned a list instead of a dict.", + 'data': result}) + raise APIError( + {'code': "Unknown API error", + 'info': "Unable to process query response of type %s." + % type(result), + 'data': result}) +

1 0

[Pywikipedia-l] [ pywikipediabot-Bugs-1825517 ] interwiki.py, redirect.py: Adding Korean
by SourceForge.net 05 Nov '07

05 Nov '07

Bugs item #1825517, was opened at 2007-11-04 13:21 Message generated for change (Settings changed) made by wikipedian You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1825517&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: interwiki Group: None >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: interwiki.py, redirect.py: Adding Korean Initial Comment: I request adding the Korean translation of bot's edit summary. interwiki.py: 'ko': (u'로봇이 ', u'더함', u'지움', u'바꿈') redirect.py: msg_double 'ko': (u'로봇: 이중 넘겨주기 수정') reason_broken 'ko': (u'로봇: 끊긴 넘겨주기') [[wikipedia:ko:User:피첼]] ---------------------------------------------------------------------- Comment By: Warddr (warddr) Date: 2007-11-04 17:45 Message: Logged In: YES user_id=1741827 Originator: NO DONE! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1825517&group_…

1 0

[Pywikipedia-l] SVN: [4506] trunk/pywikipedia/catlib.py
by erwin85＠svn.wikimedia.org 05 Nov '07

05 Nov '07

Revision: 4506 Author: erwin85 Date: 2007-11-05 10:23:03 +0000 (Mon, 05 Nov 2007) Log Message: ----------- * catlib.py fixed SpamfilterError exception Modified Paths: -------------- trunk/pywikipedia/catlib.py Modified: trunk/pywikipedia/catlib.py =================================================================== --- trunk/pywikipedia/catlib.py 2007-11-05 10:08:28 UTC (rev 4505) +++ trunk/pywikipedia/catlib.py 2007-11-05 10:23:03 UTC (rev 4506) @@ -489,7 +489,7 @@ except wikipedia.SpamfilterError, e: wikipedia.output( u'Skipping %s because of blacklist entry %s' - % (page.title(), e.url)) + % (article.title(), e.url)) except wikipedia.LockedPage: wikipedia.output( u'Skipping %s because page is locked' % article.title())

1 0

[Pywikipedia-l] SVN: [4505] trunk/pywikipedia/commonsdelinker
by btongminh＠svn.wikimedia.org 05 Nov '07

05 Nov '07

Revision: 4505 Author: btongminh Date: 2007-11-05 10:08:28 +0000 (Mon, 05 Nov 2007) Log Message: ----------- New plugins: FrWikibooks Modified Paths: -------------- trunk/pywikipedia/commonsdelinker/delinker.py Added Paths: ----------- trunk/pywikipedia/commonsdelinker/plugins/books.py Modified: trunk/pywikipedia/commonsdelinker/delinker.py =================================================================== --- trunk/pywikipedia/commonsdelinker/delinker.py 2007-11-05 07:00:05 UTC (rev 4504) +++ trunk/pywikipedia/commonsdelinker/delinker.py 2007-11-05 10:08:28 UTC (rev 4505) @@ -173,6 +173,13 @@ return 'failed' new_text = text + m_image = ImmutableByReference(image) + m_replacement = ImmutableByReference(replacement) + self.CommonsDelinker.exec_hook('before_replace', + (page, summary, m_image, m_replacement)) + image = m_image.get() + replacement = m_replacement.get() + def create_regex(s): s = re.escape(s) return ur'(?:[%s%s]%s)' % (s[0].upper(), s[0].lower(), s[1:]) @@ -190,7 +197,7 @@ if hook: if False is self.CommonsDelinker.exec_hook('%s_replace' % hook, (page, summary, image, m_replacement, match, groups)): - return u''.join(groups) + return u''.join(groups) if m_replacement.get() is None: return u'' @@ -246,7 +253,7 @@ hook = 'gallery' r_galleries = ur'(?s)(\<%s\>)(.*?)(\<\/%s\>)' % (create_regex_i('gallery'), create_regex_i('gallery')) - r_gallery = ur'(?m)^((?:%s)?)%s(\s*(?:\|.*?)?\s*)$' % (r_namespace, r_image) + r_gallery = ur'(?m)^((?:%s)?)%s(\s*(?:\|.*?)?\s*$)' % (r_namespace, r_image) def gallery_replacer(match): return ur'%s%s%s' % (match.group(1), re.sub(r_gallery, simple_replacer, match.group(2)), match.group(3)) Added: trunk/pywikipedia/commonsdelinker/plugins/books.py =================================================================== --- trunk/pywikipedia/commonsdelinker/plugins/books.py (rev 0) +++ trunk/pywikipedia/commonsdelinker/plugins/books.py 2007-11-05 10:08:28 UTC (rev 4505) @@ -0,0 +1,15 @@ +__version__ = '$Id: $' + +import re + +class FrPhotographie(object): + hook = 'before_replace' + def __init__(self, CommonsDelinker): + self.CommonsDelinker = CommonsDelinker + def __call__(self, page, summary, image, replacement): + site = page.site() + if (site.lang, site.family.name) == ('fr', 'wikibooks') and replacement.get() is None: + if page.title().startswith('Photographie/') or page.title().startswith('Tribologie/'): + replacement.set('IMG.svg') + self.CommonsDelinker.output(u'%s Replaced %s by IMG.svg on %s.' % \ + (self, image.get(), replacement.get()))

1 0

[Pywikipedia-l] SVN: [4504] trunk/pywikipedia/families/wikipedia_family.py
by siebrand＠svn.wikimedia.org 05 Nov '07

05 Nov '07

Revision: 4504 Author: siebrand Date: 2007-11-05 07:00:05 +0000 (Mon, 05 Nov 2007) Log Message: ----------- Update wikipedia_family.py Modified Paths: -------------- trunk/pywikipedia/families/wikipedia_family.py Modified: trunk/pywikipedia/families/wikipedia_family.py =================================================================== --- trunk/pywikipedia/families/wikipedia_family.py 2007-11-04 21:11:53 UTC (rev 4503) +++ trunk/pywikipedia/families/wikipedia_family.py 2007-11-05 07:00:05 UTC (rev 4504) @@ -67,6 +67,7 @@ 'hu': u'Wikipédia', 'ka': u'ვიკიპედია', 'kk': u'Уикипедия', + 'kn': u'ವಿಕಿಪೀಡಿಯ', 'ko': u'위키백과', 'ku': u'Wîkîpediya', 'la': u'Vicipaedia', @@ -159,7 +160,7 @@ 'ka': u'ვიკიპედია განხილვა', 'kab': u'Amyannan n Wikipedia', 'kk': u'Уикипедия талқылауы', - 'kn': u'Wikipedia ಚರ್ಚೆ', + 'kn': u'ವಿಕಿಪೀಡಿಯ ಚರ್ಚೆ', 'ko': u'위키백과토론', 'ksh': u'Wikipedia Klaaf', 'ku': u'Wîkîpediya nîqaş', @@ -253,7 +254,9 @@ 'sk': u'Portál', 'sr': u'Портал', 'tg': u'Портал', + 'th': u'สถานีย่อย', 'uk': u'Портал', + 'vi': u'Chủ đề', 'vo': u'Bespik dö Vükiped', 'zh-classical': u'門', } @@ -302,8 +305,10 @@ 'su': u'Obrolan portal', 'sv': u'Portaldiskussion', 'tg': u'Баҳси портал', + 'th': u'คุยเรื่องสถานีย่อย', 'tr': u'Portal tartışma', 'uk': u'Обговорення порталу', + 'vi': u'Thảo luận Chủ đề', 'zh-classical': u'議', }

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

pywikibot