I've found myself behind a proxy this week, and probably next week, and the proxy support in core is quite broken, which is slowing me down but also giving me something new to work on...
To unbreak pywikibot in my current environment, the online git repo version detection needs to be either rewritten to use a different http library, or needs to be disabled. This patch does the latter:
https://gerrit.wikimedia.org/r/#/c/137009/
The documentation for proxy support also needs to be revised:
https://gerrit.wikimedia.org/r/#/c/137273/
As I was fiddling with httplib2, I also tried to take bug 65189 which upgrades our externals to httplib2 v0.9. however, when I initiate a git review after having done a lovely merge, I see the following scary message.
You are about to submit multiple commits. This is expected if you are submitting a commit that is dependent on one or more in-review commits. Otherwise you should consider squashing your changes into one commit before submitting.
The outstanding commits are:
4ea82ce (HEAD, merge-0.9) Bug 65189: Merge upstream master (0.9) 7d1b88a (upstream/master) Create LICENSE 058a1f9 (tag: v0.9) Merge pull request #267 from jone/master c7656eb setup.py: remove download URL, update project URL dfb0d4a Fix Makefile for rename of README to README.md 84fd089 Release 0.9. The heartbleed release. 0197ec8 (tag: 0.9) Add an updated cacerts.txt file and fix some tests. Turns out nginx doesn't support etags on gzip'd content. f3d6c01 Merge pull request #259 from snarfed/master 9e1c4e5 use socket.getdefaulttimeout() on App Engine as well as off, take 2 f09a962 Merge pull request #258 from snarfed/master 3118243 use socket.getdefaulttimeout() on App Engine as well as off 1c31470 Update README.md 5277a0e whitespace fix a461c21 Remove redundant README 54ce617 Remove unused files. 13988a8 Update README.md cf958f3 Merge ../httplib2.current eabfa54 Update README.md 09d237e Last fixed for markdown. 01c687f Update README.md 2183951 Markdown fixes. c4a85f6 Moving README over to README.md df8bfce Have git and mercurial ignore each others files. b77740e Make code slightly more portable. Revewed in https://codereview.appspot.com/7406049/. c67ce1b Make indents consistent with PEP8. a6ef65e Add missing conn.close(). 2676371 Fix all whitespace issues. Patch from dhermes@google.com. fe22a15 Change the App Engine specific connection objects to be subclasses of the httplib ones, which gives better behavior on App Engine. 21fef11 Update for 0.7.7 release. 80be21f Added tag 0.7.7 for changeset e78fbc336997 63e74cf Stop swallowing App Engine specific errors. Fixes issue 230. Fixes issue 231. 1124805 Fix Python3 dup header handling. Fixes isse #229. 4fba20c Adding proxy_info_from_* methods to Python3. e078b6b Added GeoTrust cert 6982075 Make httplib2.Http() instances pickleable. 2f5763a Added tag 0.7.6 for changeset 28bf49dc86c3 06c11c0 Version 0.7.6 b3e06e2 On Python App Engine 2.7 a body can be passed in that is actually a stream. Do a full read of the contents of that stream before proceeding. 2dd5a4a Fix app engine 4a29254 Added tag 0.7.5 for changeset 403deb2f51e7 8270c14 Fix some tests. 5352702 Updating tests df935b0 Fixes issue #212. Fixes issue #210. Fixes issue #214. Fixes issue #223. Fixes issue #222. edba5f4 Added tag 0.7.5 for changeset c17f3cf9129c 6d02d69 Keys are lowercase in a Response object, regardless of how Response object is constructed. 0f0f99d Add control so that Authorization: headers aren't forwarded on a 3xx response by default. bc0b0b6 Set the reason correctly when running on App Engine. 6a5d77c Fix proxy socks for SSL connections. Fixes issue #199. 3b089c9 Add a parameter to control the number of retries. 1fa164b Fix wrong link in description. c0b0ba1 0.7.4 fix for missing cacerts f0af0ab Added tag 0.7.4 for changeset d4a0fa4d3700 43c9852 release notes 62b1798 0.7.3 tagged release 9551feb Changing version numbers for 0.7.3 release 8a66ac6 Fall back to using installed socks.py if local one is not found. 91eced9 When opening a socket for HTTP, httplib2 uses the IP and port that is returned from socket.getaddrinfo(self.host, self.port, 0, socket.SOCK_STREAM). This breaks using an HTTP CONNECT proxy by preventing the socks module from ever seeing the original domain name. Using the orignal hostname and port seem to be the correct choice here as the socket library will resolve them and it would have already been cached. 53426c1 Converted MANIFEST to manifest template (MANIFEST.in) for setuptools compatibility 0dbe275 Added support for setuptools (if present, but still fall back to distutils). Fixes issue #194 496fe69 Fixed failing test case when IPv6 HTTPS server is present 9b55a04 Fixed failing test case on Windows 5e42abd Fix python 2.4 compat issues and fixed python 3.0 tests. cf7b66f Now allow unicode to specify proxy hostname. Fixes #179 9e9e24c merged 09aa900 Added tag 0.7.2 for changeset d2d8aa1da6a7 39cb8d8 Fix makefile for generation of docs a99e87f Prepping for version 0.7.2 release 729ecab Fix up some tests. 6366968 Actually fix the issue in AppEngineHttpConnection.getresponse. 42d52b9 Raise an exception in AppEngineHttpConnection.getresponse if response is None. 19b6c4a Fixes issue 161. f9cbe4a Add socks.PROXY_TYPE_HTTP_NO_TUNNEL. Reviewed in http://codereview.appspot.com/4956072/. Fixes issue #38. fd4d5bc Fixed issue where proxy bypass detection failed when a port was specified. afff0eb no_proxy can now be upper case ProxyInfo.from_environment now honors no_proxy=* ProxyInfo.applies_to now matches the domain of the host (consistent with Python's urllib) 59d8756 Http() now detects the proxy configuration 69231c0 Added support to ProxyInfo object for detection of proxy from environment variables e65df3f Reformatted ProxyInfo for readability 130d32d Change out Go Daddy root ca for their ca bundle. Also add checks for version number matching when doing releases. 1560161 Add Go Daddy cert. Fixes issue 154. 751b173 Fix issue with not matching on first host in cert list. Add unit tests. f48303c Added (optional) opaque support to DigestAuth 5a4889c Add unit tests for app engine specific code. Fix bugs found in said tests. 8a92d73 update changelog for 0.7.1 502dc9c Added tag 0.7.1 for changeset 458615668f63 e34c25d Version 0.7.1 c9f35e6 Fixed failure to install cacerts.txt cf7a9ca Added tag 0.7.0 for changeset 54e77b3b4431 fe94ad0 Added missing html docs files 24c916a Updated docs e0a3993 Fixes issue 50. 6c50d6a More prep for 0.7.0 9652787 Starting on 0.7.0 release 23c2f8d Fixes issue 72. Always lowercase authorization header. fd66166 Fix issue 47. Redirects that become a GET should not have a body. 10df6d8 Fixes issue 19. Set Content-location on redirected HEAD requests 652c1c6 Redirect with a GET on 302 regardless of the originating method. Fixes issue 139. 14dd2f2 Fixes issue 145. 54546b5 Handle unicode in headers when writing and retrieving cache entries. Who says headers have to be ASCII! Fixes issue 138. 895175f Fixed app engine SLLCertificateError catching 732b11f Add certificate validation. Work initially started by Christoph Kern. 9d74702 Fixes issue 133. 575ee53 Set a version number. Fixes issue # 135. a3934ae Now testing against Python 3.2 aa67c09 Sync to latest version of socks.py 223983c Better fix for https on app engine 37601d3 Fix https on app engine. e52b23c Add gzip to the user-agent, in case we are making a request to an app engine project: http://code.google.com/appengine/kb/general.html#compression 621412c Uses a custom httplib shim on App Engine to wrap urlfetch, as opposed to using the standard httplib, which allows setting timeouts and checking certificates. 87a70d0 Add default support for optimistic concurrency on PATCH requests 1ae409f Fix 302 test for python3.1 c4ef467 Fix problem with ssl import in some situations. 850d92e Fixes issue 98. e829674 Fixes issue 110. aee4f66 Fixes issue 97. 8d57603 Fixes issue 120. 0abd39f Fixes issue 123. f952e7f Fixes issue 104. 756d3b3 Fixes issue 126. IPv6 under various conditions would fail. 1b828b5 Fixes issue 131. Handle socket.timeout's that occur during send. d760a1a Further fixes for issue 96. Adds tests and also fixes it for Python 3 e3e34dc Fixes issue 96. 7ad8867 proxy support: degrade gracefully when socket.socket is unavailable 37bc19f tests: misc pep8 cleanups 520cbed MANIFEST: update for python{2,3} directories ffdb086 tests: add missing __init__.py 9389957 tests: add simple functional test for proxies 84721f0 socks: many many pep8 fixes b741feb socks: clean up trailing whitespace e98b6c7 Import socks.py for proxy support from http://code.google.com/p/socksipy-branch/ 046e623 Add a simple self-contained http smoke test. e7c798c Clean up trailing whitespace. 4407548 Fix permissions on unused-header.asis 9e603da Fix for issue #94 by Leonard Richardson 4e8a2bc Fix issue #62 by trentm 3c961a0 Fix issue 91 e92cc2c Update release tags 843fe95 update changelog 916560e Fixed issue #77 bc6b460 Added to .hgignore and fixed issue #65 ae846ca Add support for Vary header
Do you really want to submit the above commits? Type 'yes' to confirm, other to cancel:
Err, no? That is 140 changesets. ;-( Surely something is wrong... ;-)
Looking at the upstream git history, they have had a few lines of concurrent development, and the current master is based on a different main line of development than the main line of development which was used for 0.8. (It looks like a hg<->git issue)
So either we pull all those changesets in, cherry pick each commit separately to continue building on the old line of development, or flatten them out into a single changeset. (roughly in order of preference)
Im guessing pulling those old changesets in through gerrit is going to create a lovely mess, so I cancelled.
How should this be done? Can someone with git commit access 'just do it' without sending 140 patches through gerrit?
Between 0.8 and 0.9 there are only two functional changesets which we need, if we wanted to only pull in these specific changesets.
https://github.com/jcgregorio/httplib2/commit/cf958f3cf3f52c752dbc4fe343127d... https://github.com/jcgregorio/httplib2/commit/0197ec868a4fc638c08358b94200ff...
The only other code changes are the following two, but they are almost certainly code paths that pywiki doesnt use.
https://github.com/jcgregorio/httplib2/commit/9e1c4e5e56009e4c08ff446b30ec3f... https://github.com/jcgregorio/httplib2/commit/3118243f371f56ba5b6cf9d7dfcea0...
-- John Vandenberg
Le 04/06/2014 10:00, John Mark Vandenberg a écrit :
The outstanding commits are:
<snip>
Do you really want to submit the above commits? Type 'yes' to confirm, other to cancel:
Hello,
The git-review utility creates a remote named 'gerrit' to push the change. Ultimately doing:
git push gerrit HEAD:refs/for/master
Your gerrit remote must be out of date, you can refresh it using either:
git remote update git fetch gerrit
That should fix the issue.
You might want to remove the 'origin' remote. Or update it to use the ssh:// url , drop the 'gerrit' remote and instruct git-review to use 'origin' as a default remote (cant remember how to set that though).
cheers,
On Jun 4, 2014 6:16 PM, "Legoktm" legoktm.wikipedia@gmail.com wrote:
On 6/4/14, 1:07 AM, Antoine Musso wrote:
Your gerrit remote must be out of date, you can refresh it using either:
Err, I don't think that's the case here. There are probably 140 commits
that are in httplib2 0.9 that weren't in 0.8.
No. There are 140 commits which stretch back to 0.7 in a separate line of development. i.e. There are two sets of commits, many of which are functionally identical but of course have different shas and parents, etc, etc.
-- John
Hi John,
On 4 June 2014 10:00, John Mark Vandenberg jayvdb@gmail.com wrote:
As I was fiddling with httplib2, I also tried to take bug 65189 which upgrades our externals to httplib2 v0.9. however, when I initiate a git review after having done a lovely merge, I see the following scary message.
(...)
Err, no? That is 140 changesets. ;-( Surely something is wrong... ;-)
Yeah, it's a bit of a mess to update :-) I should document it somewhere more durable, but what I did now:
* updated https://gerrit.wikimedia.org/r/#/admin/projects/pywikibot/externals/httplib2... to allow the 'pywikibot' group to push to 'origin' without review * on tools.pywikibot:
git clone https://github.com/jcgregorio/httplib2 httplib2-git-origin cd httplib2-git-origin git remote add gerrit ssh://gerrit/pywikibot/externals/httplib2 gitdir=$(git rev-parse --git -dir); scp -p -P 29418 pywikipedia-conversion-bot@gerrit:hooks/commit-msg ${gi tdir}/hooks/ git config user.name "tools.pywikibot" git config user.email "valhallasw@gmail.com" # gerrit is configured in .ssh/config to be gerrit.wm.o with the correct port and key
# then first update the 'origin' remote git checkout origin/master git push gerrit master:origin # this is a direct push without review
# then do the merge git fetch gerrit master git checkout FETCH_HEAD git merge origin/master git commit --amend git push gerrit HEAD:refs/for/master
So now we have https://gerrit.wikimedia.org/r/#/c/137369/ which can actually be reviewed. Ugh.
Merlijn
pywikipedia-l@lists.wikimedia.org