On Sat, December 29, 2007 1:08 am, valhallasw(a)svn.wikimedia.org wrote:
Revision: 4774
Author: valhallasw
Date: 2007-12-29 00:08:32 +0000 (Sat, 29 Dec 2007)
Log Message:
-----------
Added epydoc configuration
Added threaded cookie-eating HTTP library, based on httplib2
Added Paths:
-----------
branches/rewrite/pywikibot/data/threadedhttp.py
branches/rewrite/pywikibot/epydoc.cfg
epydoc.cfg contains a simple configuration to create epydoc documentation.
Threadedhttp.py is a wrapper around httplib2, which implements a threaded
system and cookie support.
Example code:
import logging
logging.basicConfig(level=logging.DEBUG, format='%(levelname)s
[%(threadName)s] %(message)s')
import threadedhttp
cj = threadedhttp.LockableCookieJar()
cp = threadedhttp.ConnectionPool()
import Queue
q = Queue.Queue()
for i in range(10):
threadedhttp.HttpProcessor(q,cj,cp).start()
for i in range(10):
q.put(threadedhttp.HttpRequest('http://google.com'))
q.put(threadedhttp.HttpRequest('http://en.wikipedia.org'))
for i in range(10):
q.put(None)
I have not yet come up with a good set of unit tests as most of the
functions are hard to unit-test... any suggestions on that part are very
welcome. I have implemented epydoc-style comments for all external
functions.
For the ConnectionPool: I have now implemented a connection pool with a
maximum of 5 saved concurrent connections to each host. This means that it
is possible to have up to about N(threads) connections to one host open.
We may want to change this to a maximum of 5 concurrent connections, and
locking if a connection is required that is unavailable at that time.
Secondly, no timeouts are implemented, while these probably should be
implemented: keeping connections to rarely-used servers open is useless.
Thirdly, we may want to limit the total number of connections for those
who do not have the luxury of \infty outbound connections.
Any suggestions are, as always, very welcome!
--valhallasw