https://bugzilla.wikimedia.org/show_bug.cgi?id=55889
--- Comment #3 from Merlijn van Deen valhallasw@arctus.nl --- I see. There are already some features in place, but we are maybe not using asynchronous requests at all points where it might be useful.
First of all, connections should be re-used - this is already a feature in the httplib2 library.
The next layer, comms.threadedhttp, supports asynchronous requests ('features' would be a closer term - basically, you create a request and then wait for a lock to be released). However, I don't think we use this feature anywhere, as it's not exposed in the higher-up layers.
For saving pages, which (I think) is the most relevant place for async request, we already have support, where requests that do not return a reply that has to be handled can be handled asynchronously - see Page.put_async.
For pagegenerators, we might be able to win a bit by requesting the (i+1)th page before returning the i-th page (or, for the PreloadingGenerator, by requesting the (i+1)th batch before all pages from the i-th batch have been returned).