[Pywikipedia-l] SVN: [4774] branches/rewrite/pywikibot
Merlijn van Deen
valhallasw at arctus.nl
Sat Dec 29 01:10:09 UTC 2007
On Sat, December 29, 2007 1:08 am, valhallasw at svn.wikimedia.org wrote:
> Revision: 4774
> Author: valhallasw
> Date: 2007-12-29 00:08:32 +0000 (Sat, 29 Dec 2007)
>
> Log Message:
> -----------
> Added epydoc configuration
> Added threaded cookie-eating HTTP library, based on httplib2
>
> Added Paths:
> -----------
> branches/rewrite/pywikibot/data/threadedhttp.py
> branches/rewrite/pywikibot/epydoc.cfg
epydoc.cfg contains a simple configuration to create epydoc documentation.
Threadedhttp.py is a wrapper around httplib2, which implements a threaded
system and cookie support.
Example code:
import logging
logging.basicConfig(level=logging.DEBUG, format='%(levelname)s
[%(threadName)s] %(message)s')
import threadedhttp
cj = threadedhttp.LockableCookieJar()
cp = threadedhttp.ConnectionPool()
import Queue
q = Queue.Queue()
for i in range(10):
threadedhttp.HttpProcessor(q,cj,cp).start()
for i in range(10):
q.put(threadedhttp.HttpRequest('http://google.com'))
q.put(threadedhttp.HttpRequest('http://en.wikipedia.org'))
for i in range(10):
q.put(None)
I have not yet come up with a good set of unit tests as most of the
functions are hard to unit-test... any suggestions on that part are very
welcome. I have implemented epydoc-style comments for all external
functions.
For the ConnectionPool: I have now implemented a connection pool with a
maximum of 5 saved concurrent connections to each host. This means that it
is possible to have up to about N(threads) connections to one host open.
We may want to change this to a maximum of 5 concurrent connections, and
locking if a connection is required that is unavailable at that time.
Secondly, no timeouts are implemented, while these probably should be
implemented: keeping connections to rarely-used servers open is useless.
Thirdly, we may want to limit the total number of connections for those
who do not have the luxury of \infty outbound connections.
Any suggestions are, as always, very welcome!
--valhallasw
More information about the Pywikipedia-l
mailing list