On Sat, December 29, 2007 1:08 am, valhallasw@svn.wikimedia.org wrote:

Revision: 4774 Author: valhallasw Date: 2007-12-29 00:08:32 +0000 (Sat, 29 Dec 2007)

Log Message:

Added epydoc configuration Added threaded cookie-eating HTTP library, based on httplib2

Added Paths:

branches/rewrite/pywikibot/data/threadedhttp.py
branches/rewrite/pywikibot/epydoc.cfg

epydoc.cfg contains a simple configuration to create epydoc documentation. Threadedhttp.py is a wrapper around httplib2, which implements a threaded system and cookie support.

Example code:

import logging logging.basicConfig(level=logging.DEBUG, format='%(levelname)s [%(threadName)s] %(message)s')

import threadedhttp cj = threadedhttp.LockableCookieJar() cp = threadedhttp.ConnectionPool() import Queue q = Queue.Queue()

for i in range(10): threadedhttp.HttpProcessor(q,cj,cp).start()

for i in range(10): q.put(threadedhttp.HttpRequest('http://google.com')) q.put(threadedhttp.HttpRequest('http://en.wikipedia.org'))

for i in range(10): q.put(None)

I have not yet come up with a good set of unit tests as most of the functions are hard to unit-test... any suggestions on that part are very welcome. I have implemented epydoc-style comments for all external functions.

For the ConnectionPool: I have now implemented a connection pool with a maximum of 5 saved concurrent connections to each host. This means that it is possible to have up to about N(threads) connections to one host open. We may want to change this to a maximum of 5 concurrent connections, and locking if a connection is required that is unavailable at that time. Secondly, no timeouts are implemented, while these probably should be implemented: keeping connections to rarely-used servers open is useless. Thirdly, we may want to limit the total number of connections for those who do not have the luxury of \infty outbound connections. Any suggestions are, as always, very welcome!

--valhallasw