Pywikipedia-l January 2008

pywikipedia-l@lists.wikimedia.org

22 participants
449 discussions

[ pywikipediabot-Feature Requests-1222725 ] redirect.py does not handle triple redirects
by SourceForge.net 23 Jan '08

23 Jan '08

Feature Requests item #1222725, was opened at 2005-06-17 17:31 Message generated for change (Settings changed) made by wikipedian You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1222725&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None >Status: Closed Priority: 1 Private: No Submitted By: ChongDae (cdpark) Assigned to: Nobody/Anonymous (nobody) Summary: redirect.py does not handle triple redirects Initial Comment: When page A is redirected to B, B to C, and C to D, redirect.py only links A to C. To reach D, we have to run redirect.py again. Triple (or furthur) redirect is not rare. ---------------------------------------------------------------------- Comment By: AndreasJS (andreasjs) Date: 2008-01-24 01:09 Message: Logged In: YES user_id=1738850 Originator: NO I modified redirect.py to handle multiple redirects, see patch 1877680 ---------------------------------------------------------------------- Comment By: siebrand (siebrand) Date: 2007-04-26 21:26 Message: Logged In: YES user_id=1107255 Originator: NO Please let us know if this feature request is still applicable to the current code. If no response is given, the feature request will be denied and the issue will be closed one month from now. This message was added in an effort to reduce the number of open issues on this project. Siebrand ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1222725&group_…

1 0

[ pywikipediabot-Feature Requests-1222725 ] redirect.py does not handle triple redirects
by SourceForge.net 23 Jan '08

23 Jan '08

Feature Requests item #1222725, was opened at 2005-06-17 11:31 Message generated for change (Comment added) made by andreasjs You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1222725&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Priority: 1 Private: No Submitted By: ChongDae (cdpark) Assigned to: Nobody/Anonymous (nobody) Summary: redirect.py does not handle triple redirects Initial Comment: When page A is redirected to B, B to C, and C to D, redirect.py only links A to C. To reach D, we have to run redirect.py again. Triple (or furthur) redirect is not rare. ---------------------------------------------------------------------- >Comment By: AndreasJS (andreasjs) Date: 2008-01-23 19:09 Message: Logged In: YES user_id=1738850 Originator: NO I modified redirect.py to handle multiple redirects, see patch 1877680 ---------------------------------------------------------------------- Comment By: siebrand (siebrand) Date: 2007-04-26 15:26 Message: Logged In: YES user_id=1107255 Originator: NO Please let us know if this feature request is still applicable to the current code. If no response is given, the feature request will be denied and the issue will be closed one month from now. This message was added in an effort to reduce the number of open issues on this project. Siebrand ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603141&aid=1222725&group_…

1 0

SVN: [4927] trunk/pywikipedia/wikipedia.py
by malafaya＠svn.wikimedia.org 23 Jan '08

23 Jan '08

Revision: 4927 Author: malafaya Date: 2008-01-23 17:25:16 +0000 (Wed, 23 Jan 2008) Log Message: ----------- corrected typo Modified Paths: -------------- trunk/pywikipedia/wikipedia.py Modified: trunk/pywikipedia/wikipedia.py =================================================================== --- trunk/pywikipedia/wikipedia.py 2008-01-23 15:13:54 UTC (rev 4926) +++ trunk/pywikipedia/wikipedia.py 2008-01-23 17:25:16 UTC (rev 4927) @@ -4104,7 +4104,7 @@ # No messages could be added. # We assume that the server is down. # Wait some time, then try again. - output('WARNING: No messages found it Special:Allmessages. Maybe the server is down. Retrying in %i minutes...' % retry_idle_time) + output('WARNING: No messages found in Special:Allmessages. Maybe the server is down. Retrying in %i minutes...' % retry_idle_time) time.sleep(retry_idle_time * 60) # Next time wait longer, but not longer than half an hour retry_idle_time *= 2

1 0

[ pywikipediabot-Bugs-1877535 ] Crashing on unexist language
by SourceForge.net 23 Jan '08

23 Jan '08

Bugs item #1877535, was opened at 2008-01-22 19:10 Message generated for change (Comment added) made by crochet_david You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1877535&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: Crashing on unexist language Initial Comment: Interwiki.py is crashing on projects which does not exist. I hope it skip the page for longer working. This the output when I'm working on wiktionary: Getting 1 pages from wiktionary:sco... Wiki <wikipedia._GetAll object at 0x1441cd0> does not exist yet Dump ar (wiktionary) saved Traceback (most recent call last): File "interwiki.py", line 1609, in ? bot.run() File "interwiki.py", line 1384, in run self.queryStep() File "interwiki.py", line 1363, in queryStep subj.finish(self) File "interwiki.py", line 967, in finish if self.replaceLinks(page, new, bot): File "interwiki.py", line 1094, in replaceLinks qdone = bot.oneQuery() File "interwiki.py", line 1354, in oneQuery subject.workDone(self) File "interwiki.py", line 671, in workDone iw = page.interwiki() File "/pywikipedia/wikipedia.py", line 1424, in interwiki ll = getLanguageLinks(self.get(), insite=self.site(), File "/pywikipedia/wikipedia.py", line 603, in get self._contents = self._getEditPage(get_redirect = get_redirect, throttle = throttle, sysop = sysop, nofollow_redirects=nofollow_redirects) File "/pywikipedia/wikipedia.py", line 659, in _getEditPage raise NoSuchSite(u'Wiki %s does not exist yet' % self.site()) wikipedia.NoSuchSite: Wiki wiktionary:sco does not exist yet ---------------------------------------------------------------------- Comment By: David Crochet (crochet_david) Date: 2008-01-23 16:41 Message: Logged In: YES user_id=1944800 Originator: NO Perhaps linked to a correction due to this bug, but since then the relationship interwiki which there is no account listed in user_config.py are not verified Example: in the page wikiversity:es:Plantilla:Traducción (http://es.wikiversity.org/wiki/Plantilla:Traducci%C3%B3n) linkages between projects, es, en, el, it and fr are audited (because I have an account on all these projects), however it does not check the links related to zh, ie and ca (because they are not in user_config.py). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1877535&group_…

1 0

SVN: [4926] trunk/pywikipedia/fixes.py
by wikipedian＠svn.wikimedia.org 23 Jan '08

23 Jan '08

Revision: 4926 Author: wikipedian Date: 2008-01-23 15:13:54 +0000 (Wed, 23 Jan 2008) Log Message: ----------- improvements for german grammar and date formatting Modified Paths: -------------- trunk/pywikipedia/fixes.py Modified: trunk/pywikipedia/fixes.py =================================================================== --- trunk/pywikipedia/fixes.py 2008-01-23 14:55:27 UTC (rev 4925) +++ trunk/pywikipedia/fixes.py 2008-01-23 15:13:54 UTC (rev 4926) @@ -92,10 +92,6 @@ (u'(?<!\w)(\d+|\d+[\.,]\d+)(\$|€|DM|£|¥|mg|g|kg|ml|cl|l|t|ms|min|µm|mm|cm|dm|m|km|ha|°C|kB|MB|GB|TB|W|kW|MW|GW|PS|Nm|eV|kcal|mA|mV|kV|Ω|Hz|kHz|MHz|GHz|mol|Pa|Bq|Sv|mSv)(?=\W|²|³|$)', r'\1 \2'), # Temperaturangabe mit falsch gesetztem Leerzeichen (u'(?<!\w)(\d+|\d+[\.,]\d+)° C(?=\W|²|³|$)', ur'\1 °C'), - # Kein Leerzeichen zwischen Tag und Monat - (u'(\d+)\.(Januar|Februar|März|April|Mai|Juni|Juli|August|September|Oktober|November|Dezember)', r'\1. \2'), - # Keine führende Null beim Datum - #(u'0(\d+)\. (Januar|Februar|März|April|Mai|Juni|Juli|August|September|Oktober|November|Dezember)', r'\1. \2'), # Kein Leerzeichen nach Komma (u'([a-zäöüß](\]\])?,)((\[\[)?[a-zäöüA-ZÄÖÜ])', r'\1 \3'), # Leerzeichen und Komma vertauscht @@ -353,8 +349,10 @@ #(u'†\[\[(\d)', u'† [[\\1'), #(u'&dagger;\[\[(\d)', u'† [[\\1'), (u'\[\[(\d+\. (?:Januar|Februar|März|April|Mai|Juni|Juli|August|September|Oktober|November|Dezember)) (\d{1,4})\]\]', u'[[\\1]] [[\\2]]'), - # Keine führende Null beim Datum + # Keine führende Null beim Datum (ersteinmal nur bei denen, bei denen auch ein Leerzeichen fehlt) (u'0(\d+)\.(Januar|Februar|März|April|Mai|Juni|Juli|August|September|Oktober|November|Dezember)', r'\1. \2'), + # Kein Leerzeichen zwischen Tag und Monat + (u'(\d+)\.(Januar|Februar|März|April|Mai|Juni|Juli|August|September|Oktober|November|Dezember)', r'\1. \2'), # Kein Punkt vorm Jahr (u'(\d+)\. (Januar|Februar|März|April|Mai|Juni|Juli|August|September|Oktober|November|Dezember)\.(\d{1,4})', r'\1. \2 \3'), ],

1 0

SVN: [4925] trunk/pywikipedia/checkimages.py
by filnik＠svn.wikimedia.org 23 Jan '08

23 Jan '08

Revision: 4925 Author: filnik Date: 2008-01-23 14:55:27 +0000 (Wed, 23 Jan 2008) Log Message: ----------- Adding a template to skip on commons and make a function more 'powerful' Modified Paths: -------------- trunk/pywikipedia/checkimages.py Modified: trunk/pywikipedia/checkimages.py =================================================================== --- trunk/pywikipedia/checkimages.py 2008-01-23 14:48:13 UTC (rev 4924) +++ trunk/pywikipedia/checkimages.py 2008-01-23 14:55:27 UTC (rev 4925) @@ -81,7 +81,7 @@ # That's what you want that will be added. (i.e. the {{no source}} with the right day/month/year ) n_txt = { - 'commons':'\n{{subst:nld}}', + 'commons':'\n{{subst:nld}}', 'en' :'\n{{subst:nld}}', 'it' :'\n{{subst:unverdata}}', 'ja':'{{subst:Nsd}}', @@ -89,8 +89,13 @@ 'zh' :'{{subst:No license/auto}}', } +# Text that the bot will try to see if there's already or not. If there's a +# {{ I'll use a regex to make a better check. +# This will work so: +# '{{nld' --> '\{\{(?:template:|)no[ _]license ?(?:\||\n|\}) ?' (case insensitive). +# If there's not a {{ it will work as usual (if x in Text) txt_find = { - 'commons':['{{no license', '{{nld'], + 'commons':['{{no license', '{{nld', '{{no permission since'], 'en':['{{nld', '{{no license'], 'hu':[u'{{nincsforrás',u'{{nincslicenc'], 'it':[u'{{unverdata', u'{{unverified'], @@ -866,11 +871,16 @@ wikipedia.output(u"The file description for %s is a redirect?!" % imageName ) continue for i in TextFind: - if i.lower() in g: + if '{{' in i: + regexP = re.compile('\{\{(?:template|)%s ?(?:\||\n|\}) ?' % i.split('{{')[1].replace(' ', '[ _]'), re.I) + result = regexP.findall(g) + if result != []: + tagged = True + elif i.lower() in g: tagged = True for l in hiddentemplate: if tagged == False: - res = re.findall(r'\{\{(?:[Tt]emplate:|)%s(?: \n|\||\n)' % l.lower(), g.lower()) + res = re.findall(r'\{\{(?:[Tt]emplate:|)%s(?: \n|\||\n|\})' % l.lower(), g.lower()) if res != []: #print res wikipedia.output(u'A white template found, skipping the template...')

1 0

SVN: [4924] branches/rewrite/pywikibot/data
by russblau＠svn.wikimedia.org 23 Jan '08

23 Jan '08

Revision: 4924 Author: russblau Date: 2008-01-23 14:48:13 +0000 (Wed, 23 Jan 2008) Log Message: ----------- Improve error handling, retry waits, and docstring formatting. Modified Paths: -------------- branches/rewrite/pywikibot/data/api.py branches/rewrite/pywikibot/data/http.py branches/rewrite/pywikibot/data/threadedhttp.py Modified: branches/rewrite/pywikibot/data/api.py =================================================================== --- branches/rewrite/pywikibot/data/api.py 2008-01-22 14:57:59 UTC (rev 4923) +++ branches/rewrite/pywikibot/data/api.py 2008-01-23 14:48:13 UTC (rev 4924) @@ -9,14 +9,18 @@ # __version__ = '$Id: $' - from UserDict import DictMixin -import urllib import http import simplejson as json -import warnings +import logging +import re +import traceback +import time +import urllib +lagpattern = re.compile(r"Waiting for [\d.]+: (?P<lag>\d+) seconds? lagged") + class APIError(Exception): """The wiki site returned an error message.""" def __init__(self, code, info, **kwargs): @@ -30,17 +34,26 @@ return "%(code)s: %(info)s" % self.__dict__ +class TimeoutError(Exception): + pass + + class Request(DictMixin): """A request to a Site's api.php interface. - Attributes of this object get passed as commands to api.php, and can be - get or set using the dict interface. All attributes must be strings - (unicode). Attributes supplied without values are passed to the API as - keys. + Attributes of this object (except for the special parameters listed + below) get passed as commands to api.php, and can be get or set using + the dict interface. All attributes must be strings (unicode). + Attributes supplied without values are passed to the API as keys. - @param site: The Site to which the request will be submitted. If not - supplied, uses the user's configured default Site. + @param site: The Site to which the request will be submitted. If not + supplied, uses the user's configured default Site. @param format: (optional) Defaults to "json" + @param max_retries: (optional) Maximum number of times to retry after + errors, defaults to 25 + @param retry_wait: (optional) Minimum time to wait after an error, + defaults to 5 seconds (doubles each retry until max of 120 is + reached) Example: @@ -60,13 +73,15 @@ """ def __init__(self, *args, **kwargs): - if "site" in kwargs: - self.site = kwargs["site"] - del kwargs["site"] + self.site = kwargs.pop("site", None) # else use defaultSite() ... when written + self.max_retries = kwargs.pop("max_retries", 25) + self.retry_wait = kwargs.pop("retry_wait", 5) self.params = {} - if not "format" in kwargs: + if "format" not in kwargs: self.params["format"] = "json" + if "maxlag" not in kwargs: + self.params["maxlag"] = "5" self.update(*args, **kwargs) # implement dict interface @@ -101,7 +116,7 @@ def submit(self): """Submit a query and parse the response. - @return: The data retrieved from api.php (a dict) + @return: The data retrieved from api.php (a dict) """ if self.params['format'] != 'json': @@ -112,52 +127,70 @@ while True: # TODO wait on errors # TODO catch http errors - if self.params.get("action", "") in ("login",): - rawdata = http.request(self.site, uri, method="POST", + try: + if self.params.get("action", "") in ("login",): + rawdata = http.request(self.site, uri, method="POST", headers={'Content-Type': - 'application/x-www-form-urlencoded'}, + 'application/x-www-form-urlencoded'}, body=params) - return rawdata - else: - uri = uri + "?" + params - rawdata = http.request(self.site, uri) + else: + uri = uri + "?" + params + rawdata = http.request(self.site, uri) + except Exception, e: #TODO: what exceptions can occur here? + logging.warning(traceback.format_exc()) + self.wait() + continue if rawdata.startswith(u"unknown_action"): - e = {'code': data[:14], 'info': data[16:]} - raise APIError(e) + raise APIError(rawdata[:14], rawdata[16:]) try: result = json.loads(rawdata) except ValueError: # if the result isn't valid JSON, there must be a server # problem. Wait a few seconds and try again # TODO: implement a throttle - warnings.warn( + logging.warning( "Non-JSON response received from server %s; the server may be down." % self.site) print rawdata + self.wait(max_retries, retry_wait) continue if not result: - return {} - if type(result) is dict: - if "error" in result: - if "code" in result["error"]: - code = result["error"]["code"] - del result["error"]["code"] - else: - code = "Unknown" - if "info" in result["error"]: - info = result["error"]["info"] - del result["error"]["info"] - else: - info = None - # raise error - raise APIError(code, info, **result["error"]) + result = {} + if type(result) is not dict: + raise APIError("Unknown", + "Unable to process query response of type %s." + % type(result), + {'data': result}) + if "error" not in result: return result - raise APIError("Unknown", - "Unable to process query response of type %s." - % type(result), - {'data': result}) + code = result["error"].pop("code", "Unknown") + info = result["error"].pop("info", None) + if code == "maxlag": + lag = lagpattern.search(info) + if lag: + logging.info( + "Pausing due to database lag: " + info) + self.wait(int(lag.group("lag"))) + continue + # raise error + raise APIError(code, info, **result["error"]) + + def wait(self, lag=None): + """Determine how long to wait after a failed request.""" + self.max_retries -= 1 + if self.max_retries < 0: + raise TimeoutError("Maximum retries attempted without success.") + + if lag is not None: + if lag > 2 * self.retry_wait: + self.retry_wait = min(120, lag // 2) + logging.warn("Waiting %s seconds before retrying." % self.retry_wait) + time.sleep(self.retry_wait) + self.retry_wait = min(120, self.retry_wait * 2) + + if __name__ == "__main__": from pywikibot.tests.dummy import TestSite as Site mysite = Site("en.wikipedia.org") - + logging.getLogger().setLevel(logging.DEBUG) Modified: branches/rewrite/pywikibot/data/http.py =================================================================== --- branches/rewrite/pywikibot/data/http.py 2008-01-22 14:57:59 UTC (rev 4923) +++ branches/rewrite/pywikibot/data/http.py 2008-01-23 14:48:13 UTC (rev 4924) @@ -5,11 +5,12 @@ This module handles communication between the bot and the HTTP threads. This module is responsible for - * Setting up a connection pool - * Providing a (blocking) interface for HTTP requests - * Translate site objects with query strings into urls - * Urlencoding all data - * Basic HTTP error handling + +- Setting up a connection pool +- Providing a (blocking) interface for HTTP requests +- Translate site objects with query strings into urls +- Urlencoding all data +- Basic HTTP error handling """ # @@ -68,7 +69,14 @@ request = threadedhttp.HttpRequest(uri, *args, **kwargs) http_queue.put(request) request.lock.acquire() - + #do some error correcting stuff - + + #if all else fails + if isinstance(request.data, Exception): + raise request.data + + if request.data[0].status != 200: + logging.warning("Http response status %s" % request.data[0].status) + return request.data[1] Modified: branches/rewrite/pywikibot/data/threadedhttp.py =================================================================== --- branches/rewrite/pywikibot/data/threadedhttp.py 2008-01-22 14:57:59 UTC (rev 4923) +++ branches/rewrite/pywikibot/data/threadedhttp.py 2008-01-23 14:48:13 UTC (rev 4924) @@ -1,10 +1,12 @@ -# -*- coding: utf-8 -*- +# -*- coding: utf-8 -*- """ Httplib2 threaded cookie layer - This class extends Httplib2, adding support for: - * Cookies, guarded for cross-site redirects - * Thread safe ConnectionPool and LockableCookieJar classes - * HttpProcessor thread class - * HttpRequest object + +This class extends Httplib2, adding support for: +- Cookies, guarded for cross-site redirects +- Thread safe ConnectionPool and LockableCookieJar classes +- HttpProcessor thread class +- HttpRequest object + """ # (C) 2007 Pywikipedia bot team, 2007 @@ -34,17 +36,19 @@ class ConnectionPool(object): - """ A thread-safe connection pool """ + """A thread-safe connection pool. + + @param maxnum: Maximum number of connections per identifier. + The pool drops excessive connections added. + + """ def __init__(self, maxnum=5): - """ @param maxnum: Maximum number of connections per identifier. - The pool drops excessive connections added. - """ self.connections = {} self.lock = threading.Lock() self.maxnum = maxnum def __del__(self): - """ Destructor to close all connections in the pool """ + """Destructor to close all connections in the pool.""" self.lock.acquire() try: for key in self.connections: @@ -57,9 +61,11 @@ return self.connections.__repr__() def pop_connection(self, identifier): - """ Gets a connection from identifiers connection pool - @param identifier The pool identifier - @returns A connection object if found, None otherwise + """Get a connection from identifier's connection pool. + + @param identifier: The pool identifier + @return: A connection object if found, None otherwise + """ self.lock.acquire() try: @@ -71,9 +77,11 @@ self.lock.release() def push_connection(self, identifier, connection): - """ Adds a connection to identifiers connection pool - @param identifier The pool identifier - @param connection The connection to add to the pool + """Add a connection to identifier's connection pool. + + @param identifier: The pool identifier + @param connection: The connection to add to the pool + """ self.lock.acquire() try: @@ -96,31 +104,42 @@ self.lock = threading.Lock() class Http(httplib2.Http): - """ Subclass of httplib2.Http that uses a `LockableCookieJar` to store cookies. - Overrides httplib2s internal redirect support to prevent cookies - being eaten by the wrong sites. + """Subclass of httplib2.Http that stores cookies. + + Overrides httplib2's internal redirect support to prevent cookies being + eaten by the wrong sites. + + @param cookiejar: (optional) CookieJar to use. A new one will be used + when not supplied. + @param connection_pool: (optional) Connection pool to use. A new one + will be used when not supplied. + @param max_redirects: (optional) The maximum number of redirects to + follow. 5 is default. + """ def __init__(self, *args, **kwargs): - """ @param cookiejar: (optional) CookieJar to use. A new one will be used when not supplied. - @param connection_pool: (optional) Connection pool to use. A new one will be used when not supplied. - @param max_redirects: (optional) The maximum number of redirects to follow. 5 is default. - """ self.cookiejar = kwargs.pop('cookiejar', LockableCookieJar()) self.connection_pool = kwargs.pop('connection_pool', ConnectionPool()) self.max_redirects = kwargs.pop('max_redirects', 5) httplib2.Http.__init__(self, *args, **kwargs) - def request(self, uri, method="GET", body=None, headers=None, max_redirects=None, connection_type=None): - """ Starts an HTTP request. - @param uri: The uri to retrieve - @param method: (optional) The HTTP method to use. Default is 'GET' - @param body: (optional) The request body. Default is no body. - @param headers: (optional) Additional headers to send. Defaults include - C{connection: keep-alive}, C{user-agent} and C{content-type}. - @param max_redirects: (optional) The maximum number of redirects to use for this request. - The class instances max_redirects is default - @param connection_type: (optional) ? - @returns: (response, content) tuple + def request(self, uri, method="GET", body=None, headers=None, + max_redirects=None, connection_type=None): + """Start an HTTP request. + + @param uri: The uri to retrieve + @param method: (optional) The HTTP method to use. Default is 'GET' + @param body: (optional) The request body. Default is no body. + @param headers: (optional) Additional headers to send. Defaults + include C{connection: keep-alive}, C{user-agent} and + C{content-type}. + @param max_redirects: (optional) The maximum number of redirects to + use for this request. The class instance's max_redirects is + default + @param connection_type: (optional) see L{httplib2.Http.request} + + @return: (response, content) tuple + """ if max_redirects is None: max_redirects = self.max_redirects @@ -136,11 +155,13 @@ self.cookiejar.lock.release() headers = req.headers - # Wikimedia squids: add connection: keep-alive to request headers unless overridden + # Wikimedia squids: add connection: keep-alive to request headers + # unless overridden headers['connection'] = headers.pop('connection', 'keep-alive') # determine connection pool key and fetch connection - (scheme, authority, request_uri, defrag_uri) = httplib2.urlnorm(httplib2.iri2uri(uri)) + (scheme, authority, request_uri, defrag_uri) = httplib2.urlnorm( + httplib2.iri2uri(uri)) conn_key = scheme+":"+authority connection = self.connection_pool.pop_connection(conn_key) @@ -150,12 +171,20 @@ # Redirect hack: we want to regulate redirects follow_redirects = self.follow_redirects self.follow_redirects = False - logging.debug('%r' % ((uri, method, headers, max_redirects, connection_type),)) - (response, content) = httplib2.Http.request(self, uri, method, body, headers, max_redirects, connection_type) + logging.debug('%r' % ( + (uri, method, body, headers, max_redirects, connection_type),)) + try: + (response, content) = httplib2.Http.request( + self, uri, method, body, headers, + max_redirects, connection_type) + except Exception, e: # what types? + # return exception instance to be retrieved by the calling thread + return e self.follow_redirects = follow_redirects # return connection to pool - self.connection_pool.push_connection(conn_key, self.connections[conn_key]) + self.connection_pool.push_connection(conn_key, + self.connections[conn_key]) del self.connections[conn_key] # First write cookies @@ -167,15 +196,20 @@ # Check for possible redirects redirectable_response = ((response.status == 303) or - (response.status in [300, 301, 302, 307] and method in ["GET", "HEAD"])) - if self.follow_redirects and (max_redirects > 0) and redirectable_response: - (response, content) = self._follow_redirect(uri, method, body, headers, response, content, max_redirects) + (response.status in [300, 301, 302, 307] and + method in ["GET", "HEAD"])) + if self.follow_redirects and (max_redirects > 0) \ + and redirectable_response: + (response, content) = self._follow_redirect( + uri, method, body, headers, response, content, max_redirects) return (response, content) - def _follow_redirect(self, uri, method, body, headers, response, content, max_redirects): - """ Internal function to follow a redirect recieved by L{request} """ - (scheme, authority, absolute_uri, defrag_uri) = httplib2.urlnorm(httplib2.iri2uri(uri)) + def _follow_redirect(self, uri, method, body, headers, response, + content, max_redirects): + """Internal function to follow a redirect recieved by L{request}""" + (scheme, authority, absolute_uri, defrag_uri) = httplib2.urlnorm( + httplib2.iri2uri(uri)) if self.cache: cachekey = defrag_uri else: @@ -184,39 +218,54 @@ # Pick out the location header and basically start from the beginning # remembering first to strip the ETag header and decrement our 'depth' if not response.has_key('location') and response.status != 300: - raise httplib2.RedirectMissingLocation("Redirected but the response is missing a Location: header.", response, content) + raise httplib2.RedirectMissingLocation( + "Redirected but the response is missing a Location: header.", + response, content) # Fix-up relative redirects (which violate an RFC 2616 MUST) if response.has_key('location'): location = response['location'] - (scheme, authority, path, query, fragment) = httplib2.parse_uri(location) + (scheme, authority, path, query, fragment) = httplib2.parse_uri( + location) if authority == None: response['location'] = httplib2.urlparse.urljoin(uri, location) - logging.debug('Relative redirect: changed [%s] to [%s]' % (location, response['location'])) + logging.debug('Relative redirect: changed [%s] to [%s]' + % (location, response['location'])) if response.status == 301 and method in ["GET", "HEAD"]: response['-x-permanent-redirect-url'] = response['location'] if not response.has_key('content-location'): response['content-location'] = absolute_uri - httplib2._updateCache(headers, response, content, self.cache, cachekey) + httplib2._updateCache(headers, response, content, self.cache, + cachekey) headers.pop('if-none-match', None) headers.pop('if-modified-since', None) if response.has_key('location'): location = response['location'] - redirect_method = ((response.status == 303) and (method not in ["GET", "HEAD"])) and "GET" or method - return self.request(location, redirect_method, body=body, headers = headers, max_redirects = max_redirects - 1) + redirect_method = ((response.status == 303) and + (method not in ["GET", "HEAD"]) + ) and "GET" or method + return self.request(location, redirect_method, body=body, + headers=headers, + max_redirects=max_redirects - 1) else: - raise RedirectLimit("Redirected more times than redirection_limit allows.", response, content) + raise RedirectLimit( + "Redirected more times than redirection_limit allows.", + response, content) + class HttpRequest(object): - """ Object wrapper for HTTP requests that need to block the requesters thread. - Usage: - >>> request = HttpRequest('http://www.google.com') - >>> queue.put(request) - >>> request.lock.acquire() - >>> print request.data - - C{request.lock.acquire()} will block until the data is available. + """Object wrapper for HTTP requests that need to block origin thread. + + Usage: + + >>> request = HttpRequest('http://www.google.com') + >>> queue.put(request) + >>> request.lock.acquire() + >>> print request.data + + C{request.lock.acquire()} will block until the data is available. + """ def __init__(self, *args, **kwargs): self.args = args @@ -224,13 +273,19 @@ self.data = None self.lock = threading.Semaphore(0) + class HttpProcessor(threading.Thread): - """ Thread object to spawn multiple HTTP connection threads """ + """ Thread object to spawn multiple HTTP connection threads. + + @param queue: The C{Queue.Queue} object that contains L{HttpRequest} + objects. + @param cookiejar: The C{LockableCookieJar} cookie object to share among + requests. + @param connection_pool: The C{ConnectionPool} object which contains + connections to share among requests. + + """ def __init__(self, queue, cookiejar, connection_pool): - """ @param queue: The C{Queue.Queue} object that contains L{HttpRequest} objects. - @param cookiejar: The C{LockableCookieJar} cookie object to share among requests. - @param connection_pool: The C{ConnectionPool} object which contains connections to share among requests. - """ threading.Thread.__init__(self) self.queue = queue self.http = Http(cookiejar=cookiejar, connection_pool=connection_pool) @@ -352,4 +407,4 @@ # as part of the expires= date format. so we have # to split carefully here - header.split(',') won't do it. HEADERVAL= re.compile(r'\s*(([^,]|(,\s*\d))+)') - return [h[0] for h in HEADERVAL.findall(self.response[k])] \ No newline at end of file + return [h[0] for h in HEADERVAL.findall(self.response[k])]

1 0

[ pywikipediabot-Patches-1877680 ] redirect.py multiple redirects
by SourceForge.net 22 Jan '08

22 Jan '08

Patches item #1877680, was opened at 2008-01-22 16:57 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1877680&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: AndreasJS (andreasjs) Assigned to: Nobody/Anonymous (nobody) Summary: redirect.py multiple redirects Initial Comment: Will handle multiple redirects. Detects and skips recursive redirects. Follows into other sites (interactive mode only). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603140&aid=1877680&group_…

1 0

[ pywikipediabot-Bugs-1877535 ] Crashing on unexist language
by SourceForge.net 22 Jan '08

22 Jan '08

Bugs item #1877535, was opened at 2008-01-22 10:10 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1877535&group_… Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nobody/Anonymous (nobody) Assigned to: Nobody/Anonymous (nobody) Summary: Crashing on unexist language Initial Comment: Interwiki.py is crashing on projects which does not exist. I hope it skip the page for longer working. This the output when I'm working on wiktionary: Getting 1 pages from wiktionary:sco... Wiki <wikipedia._GetAll object at 0x1441cd0> does not exist yet Dump ar (wiktionary) saved Traceback (most recent call last): File "interwiki.py", line 1609, in ? bot.run() File "interwiki.py", line 1384, in run self.queryStep() File "interwiki.py", line 1363, in queryStep subj.finish(self) File "interwiki.py", line 967, in finish if self.replaceLinks(page, new, bot): File "interwiki.py", line 1094, in replaceLinks qdone = bot.oneQuery() File "interwiki.py", line 1354, in oneQuery subject.workDone(self) File "interwiki.py", line 671, in workDone iw = page.interwiki() File "/pywikipedia/wikipedia.py", line 1424, in interwiki ll = getLanguageLinks(self.get(), insite=self.site(), File "/pywikipedia/wikipedia.py", line 603, in get self._contents = self._getEditPage(get_redirect = get_redirect, throttle = throttle, sysop = sysop, nofollow_redirects=nofollow_redirects) File "/pywikipedia/wikipedia.py", line 659, in _getEditPage raise NoSuchSite(u'Wiki %s does not exist yet' % self.site()) wikipedia.NoSuchSite: Wiki wiktionary:sco does not exist yet ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=603138&aid=1877535&group_…

1 0

SVN: [4923] trunk/pywikipedia/add_text.py
by filnik＠svn.wikimedia.org 22 Jan '08

22 Jan '08

Revision: 4923 Author: filnik Date: 2008-01-22 14:57:59 +0000 (Tue, 22 Jan 2008) Log Message: ----------- Bugfix, replaceCategoryLinks() will only add the category, not replace them Modified Paths: -------------- trunk/pywikipedia/add_text.py Modified: trunk/pywikipedia/add_text.py =================================================================== --- trunk/pywikipedia/add_text.py 2008-01-22 14:57:27 UTC (rev 4922) +++ trunk/pywikipedia/add_text.py 2008-01-22 14:57:59 UTC (rev 4923) @@ -174,7 +174,7 @@ # Adding the text newtext += u"\n%s" % addText # Reputting the categories - newtext = wikipedia.replaceCategoryLinks(newtext, categoriesInside, site) + newtext = wikipedia.replaceCategoryLinks(newtext, categoriesInside, site, True) # Dealing the stars' issue starsListInPage = list() for star in starsList: @@ -189,7 +189,7 @@ # If instead the text must be added above... else: newtext = addText + '\n' + text - if putText: + if putText and text != newtext: wikipedia.output(u"\n\n>>> \03{lightpurple}%s\03{default} <<<" % page.title()) wikipedia.showDiff(text, newtext) choice = ''

1 0

← Newer
1
...
8
9
10
11
12
13
14
...
45
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

Pywikipedia-l January 2008