I've found myself behind a proxy this week, and probably next week,
and the proxy support in core is quite broken, which is slowing me
down but also giving me something new to work on...
To unbreak pywikibot in my current environment, the online git repo
version detection needs to be either rewritten to use a different http
library, or needs to be disabled. This patch does the latter:
https://gerrit.wikimedia.org/r/#/c/137009/
The documentation for proxy support also needs to be revised:
https://gerrit.wikimedia.org/r/#/c/137273/
As I was fiddling with httplib2, I also tried to take bug 65189 which
upgrades our externals to httplib2 v0.9. however, when I initiate a
git review after having done a lovely merge, I see the following scary
message.
You are about to submit multiple commits. This is expected if you are
submitting a commit that is dependent on one or more in-review
commits. Otherwise you should consider squashing your changes into one
commit before submitting.
The outstanding commits are:
4ea82ce (HEAD, merge-0.9) Bug 65189: Merge upstream master (0.9)
7d1b88a (upstream/master) Create LICENSE
058a1f9 (tag: v0.9) Merge pull request #267 from jone/master
c7656eb setup.py: remove download URL, update project URL
dfb0d4a Fix Makefile for rename of README to README.md
84fd089 Release 0.9. The heartbleed release.
0197ec8 (tag: 0.9) Add an updated cacerts.txt file and fix some tests.
Turns out nginx doesn't support etags on gzip'd content.
f3d6c01 Merge pull request #259 from snarfed/master
9e1c4e5 use socket.getdefaulttimeout() on App Engine as well as off, take 2
f09a962 Merge pull request #258 from snarfed/master
3118243 use socket.getdefaulttimeout() on App Engine as well as off
1c31470 Update README.md
5277a0e whitespace fix
a461c21 Remove redundant README
54ce617 Remove unused files.
13988a8 Update README.md
cf958f3 Merge ../httplib2.current
eabfa54 Update README.md
09d237e Last fixed for markdown.
01c687f Update README.md
2183951 Markdown fixes.
c4a85f6 Moving README over to README.md
df8bfce Have git and mercurial ignore each others files.
b77740e Make code slightly more portable. Revewed in
https://codereview.appspot.com/7406049/.
c67ce1b Make indents consistent with PEP8.
a6ef65e Add missing conn.close().
2676371 Fix all whitespace issues. Patch from dhermes(a)google.com.
fe22a15 Change the App Engine specific connection objects to be
subclasses of the httplib ones, which gives better behavior on App
Engine.
21fef11 Update for 0.7.7 release.
80be21f Added tag 0.7.7 for changeset e78fbc336997
63e74cf Stop swallowing App Engine specific errors. Fixes issue 230.
Fixes issue 231.
1124805 Fix Python3 dup header handling. Fixes isse #229.
4fba20c Adding proxy_info_from_* methods to Python3.
e078b6b Added GeoTrust cert
6982075 Make httplib2.Http() instances pickleable.
2f5763a Added tag 0.7.6 for changeset 28bf49dc86c3
06c11c0 Version 0.7.6
b3e06e2 On Python App Engine 2.7 a body can be passed in that is
actually a stream. Do a full read of the contents of that stream
before proceeding.
2dd5a4a Fix app engine
4a29254 Added tag 0.7.5 for changeset 403deb2f51e7
8270c14 Fix some tests.
5352702 Updating tests
df935b0 Fixes issue #212. Fixes issue #210. Fixes issue #214. Fixes
issue #223. Fixes issue #222.
edba5f4 Added tag 0.7.5 for changeset c17f3cf9129c
6d02d69 Keys are lowercase in a Response object, regardless of how
Response object is constructed.
0f0f99d Add control so that Authorization: headers aren't forwarded on
a 3xx response by default.
bc0b0b6 Set the reason correctly when running on App Engine.
6a5d77c Fix proxy socks for SSL connections. Fixes issue #199.
3b089c9 Add a parameter to control the number of retries.
1fa164b Fix wrong link in description.
c0b0ba1 0.7.4 fix for missing cacerts
f0af0ab Added tag 0.7.4 for changeset d4a0fa4d3700
43c9852 release notes
62b1798 0.7.3 tagged release
9551feb Changing version numbers for 0.7.3 release
8a66ac6 Fall back to using installed socks.py if local one is not found.
91eced9 When opening a socket for HTTP, httplib2 uses the IP and port
that is returned from socket.getaddrinfo(self.host, self.port, 0,
socket.SOCK_STREAM). This breaks using an HTTP CONNECT proxy by
preventing the socks module from ever seeing the original domain name.
Using the orignal hostname and port seem to be the correct choice here
as the socket library will resolve them and it would have already been
cached.
53426c1 Converted MANIFEST to manifest template (MANIFEST.in) for
setuptools compatibility
0dbe275 Added support for setuptools (if present, but still fall back
to distutils). Fixes issue #194
496fe69 Fixed failing test case when IPv6 HTTPS server is present
9b55a04 Fixed failing test case on Windows
5e42abd Fix python 2.4 compat issues and fixed python 3.0 tests.
cf7b66f Now allow unicode to specify proxy hostname. Fixes #179
9e9e24c merged
09aa900 Added tag 0.7.2 for changeset d2d8aa1da6a7
39cb8d8 Fix makefile for generation of docs
a99e87f Prepping for version 0.7.2 release
729ecab Fix up some tests.
6366968 Actually fix the issue in AppEngineHttpConnection.getresponse.
42d52b9 Raise an exception in AppEngineHttpConnection.getresponse if
response is None.
19b6c4a Fixes issue 161.
f9cbe4a Add socks.PROXY_TYPE_HTTP_NO_TUNNEL. Reviewed in
http://codereview.appspot.com/4956072/. Fixes issue #38.
fd4d5bc Fixed issue where proxy bypass detection failed when a port
was specified.
afff0eb no_proxy can now be upper case ProxyInfo.from_environment now
honors no_proxy=* ProxyInfo.applies_to now matches the domain of the
host (consistent with Python's urllib)
59d8756 Http() now detects the proxy configuration
69231c0 Added support to ProxyInfo object for detection of proxy from
environment variables
e65df3f Reformatted ProxyInfo for readability
130d32d Change out Go Daddy root ca for their ca bundle. Also add
checks for version number matching when doing releases.
1560161 Add Go Daddy cert. Fixes issue 154.
751b173 Fix issue with not matching on first host in cert list. Add unit tests.
f48303c Added (optional) opaque support to DigestAuth
5a4889c Add unit tests for app engine specific code. Fix bugs found in
said tests.
8a92d73 update changelog for 0.7.1
502dc9c Added tag 0.7.1 for changeset 458615668f63
e34c25d Version 0.7.1
c9f35e6 Fixed failure to install cacerts.txt
cf7a9ca Added tag 0.7.0 for changeset 54e77b3b4431
fe94ad0 Added missing html docs files
24c916a Updated docs
e0a3993 Fixes issue 50.
6c50d6a More prep for 0.7.0
9652787 Starting on 0.7.0 release
23c2f8d Fixes issue 72. Always lowercase authorization header.
fd66166 Fix issue 47. Redirects that become a GET should not have a body.
10df6d8 Fixes issue 19. Set Content-location on redirected HEAD requests
652c1c6 Redirect with a GET on 302 regardless of the originating
method. Fixes issue 139.
14dd2f2 Fixes issue 145.
54546b5 Handle unicode in headers when writing and retrieving cache
entries. Who says headers have to be ASCII! Fixes issue 138.
895175f Fixed app engine SLLCertificateError catching
732b11f Add certificate validation. Work initially started by Christoph Kern.
9d74702 Fixes issue 133.
575ee53 Set a version number. Fixes issue # 135.
a3934ae Now testing against Python 3.2
aa67c09 Sync to latest version of socks.py
223983c Better fix for https on app engine
37601d3 Fix https on app engine.
e52b23c Add gzip to the user-agent, in case we are making a request to
an app engine project:
http://code.google.com/appengine/kb/general.html#compression
621412c Uses a custom httplib shim on App Engine to wrap urlfetch, as
opposed to using the standard httplib, which allows setting timeouts
and checking certificates.
87a70d0 Add default support for optimistic concurrency on PATCH requests
1ae409f Fix 302 test for python3.1
c4ef467 Fix problem with ssl import in some situations.
850d92e Fixes issue 98.
e829674 Fixes issue 110.
aee4f66 Fixes issue 97.
8d57603 Fixes issue 120.
0abd39f Fixes issue 123.
f952e7f Fixes issue 104.
756d3b3 Fixes issue 126. IPv6 under various conditions would fail.
1b828b5 Fixes issue 131. Handle socket.timeout's that occur during send.
d760a1a Further fixes for issue 96. Adds tests and also fixes it for Python 3
e3e34dc Fixes issue 96.
7ad8867 proxy support: degrade gracefully when socket.socket is unavailable
37bc19f tests: misc pep8 cleanups
520cbed MANIFEST: update for python{2,3} directories
ffdb086 tests: add missing __init__.py
9389957 tests: add simple functional test for proxies
84721f0 socks: many many pep8 fixes
b741feb socks: clean up trailing whitespace
e98b6c7 Import socks.py for proxy support from
http://code.google.com/p/socksipy-branch/
046e623 Add a simple self-contained http smoke test.
e7c798c Clean up trailing whitespace.
4407548 Fix permissions on unused-header.asis
9e603da Fix for issue #94 by Leonard Richardson
4e8a2bc Fix issue #62 by trentm
3c961a0 Fix issue 91
e92cc2c Update release tags
843fe95 update changelog
916560e Fixed issue #77
bc6b460 Added to .hgignore and fixed issue #65
ae846ca Add support for Vary header
Do you really want to submit the above commits?
Type 'yes' to confirm, other to cancel:
Err, no?
That is 140 changesets. ;-(
Surely something is wrong... ;-)
Looking at the upstream git history, they have had a few lines of
concurrent development, and the current master is based on a different
main line of development than the main line of development which was
used for 0.8. (It looks like a hg<->git issue)
So either we pull all those changesets in, cherry pick each commit
separately to continue building on the old line of development, or
flatten them out into a single changeset. (roughly in order of
preference)
Im guessing pulling those old changesets in through gerrit is going to
create a lovely mess, so I cancelled.
How should this be done? Can someone with git commit access 'just do
it' without sending 140 patches through gerrit?
Between 0.8 and 0.9 there are only two functional changesets which we
need, if we wanted to only pull in these specific changesets.
https://github.com/jcgregorio/httplib2/commit/cf958f3cf3f52c752dbc4fe343127…https://github.com/jcgregorio/httplib2/commit/0197ec868a4fc638c08358b94200f…
The only other code changes are the following two, but they are almost
certainly code paths that pywiki doesnt use.
https://github.com/jcgregorio/httplib2/commit/9e1c4e5e56009e4c08ff446b30ec3…https://github.com/jcgregorio/httplib2/commit/3118243f371f56ba5b6cf9d7dfcea…
--
John Vandenberg
I am not sure it's always better to use site.code instead of site.lang.
If TranslateWiki uses language codes, 'be-tarask' should be used instead
of 'be-x-old', etc.
You may also omit the script path, it is default. And the .py ending of the script may also be omitted. Just call pwb.py xxx
followed by any options behind that.
Best
xqt
----- Ursprüngliche Nachricht -----
Von: John Mark Vandenberg
Gesendet: 03.06.2014 06:05
An: Pywikipedia discussion list
Betreff: Re: [Pywikipedia-l] Deprecation of compat
On Mon, Jun 2, 2014 at 8:00 PM, Amir Ladsgroup <ladsgroup(a)gmail.com> wrote:
> Core has one million advantages on compat but easy-installation is not one
> of them, a week ago I was trying to teach some bot operators to migrate to
> core and there were three main problems:
>
> 1- core suggests people to make the user-config in a .pywikibot folder
> and people don't like this and changing it isn't easy (for example if you
> want to set it as current folder, like compat, which is a common act because
> people use several accounts and they need several pywikibot folders with
> separated user-config and the system doesn't let you to make a file in
> existing director, try doing this)
This part of the startup is quite problematic, as the 'config
directory' has three different sets of logic
1. pwb.py
2. generate_user_files.py get_base_dir()
3. pywikibot/config2.py _get_base_dir()
https://gerrit.wikimedia.org/r/#/c/87893/ tries to copy the logic in
config2.py into pwb.py, but really we need one function that is reused
in all three cases.
> 2- people don't know what command is best for installation:
> python pwb.py
> python generate_user_files.py
> python setup.py install
> python ez_setup.py
IMO we should promote 'python pwb.py' as the way that people start using pywiki.
> 3- There is an extra external needed (httplib2) and there is no
> documentation (in files) at all about them and how we can install them, I
> learned about them by experience and this command won't work: (my friend
> told me, I haven't checked it yet)
>
> git submodule update --init
I quickly tested this , and it appears to be working correctly when
used with 'python pwb.py scripts/xxxx.py'
--
John Vandenberg
_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
Indeed documentation ist that part of code which coders does mostly neglects.
Anyway there are some hints:
README.rst in core Folder
https://www.google.de/#q=pwb.py+site:mediawiki.org
Seems our manual at mediawiki.org needs some cleanups. We had some discussions about this theme few days ago.
Best
xqt
----- Original Nachricht ----
Von: Bináris <wikiposta(a)gmail.com>
An: Xqt <info(a)gno.de>
Datum: 02.06.2014 09:11
Betreff: Re: Re: [Pywikipedia-l] Deprecation of compat
> 2014-06-02 9:01 GMT+02:00 <info(a)gno.de>:
>
> > As I wroth with my last mail this behaviour is available since 3th june
> > 2012 using the pwb.py wrapper script.
> >
>
> That wasn't quite clear for me at once, but thnak you, this is a great
> news!
>
Seems nobody RTFM :7
greetings
xqt
----- Ursprüngliche Nachricht -----
Von: Amir Ladsgroup
Gesendet: 02.06.2014 12:01
An: Pywikipedia discussion list
Betreff: Re: [Pywikipedia-l] Deprecation of compat
Core has one million advantages on compat but easy-installation is not one of them, a week ago I was trying to teach some bot operators to migrate to core and there were three main problems:
1- core suggests people to make the user-config in a .pywikibot folder
and people don't like this and changing it isn't easy (for example if you want to set it as current folder, like compat, which is a common act because people use several accounts and they need several pywikibot folders with separated user-config and the system doesn't let you to make a file in existing director, try doing this)
2- people don't know what command is best for installation:
python pwb.py
python generate_user_files.py
python setup.py install
python ez_setup.py
3- There is an extra external needed (httplib2) and there is no documentation (in files) at all about them and how we can install them, I learned about them by experience and this command won't work: (my friend told me, I haven't checked it yet)
git submodule update --init
Helping to fix this would fix lots of user-friendliness issues
Best
On Mon, Jun 2, 2014 at 12:47 PM, <info(a)gno.de> wrote:
Indeed documentation ist that part of code which coders does mostly neglects.
Anyway there are some hints:
README.rst in core Folder
https://www.google.de/#q=pwb.py+site:mediawiki.org
Seems our manual at mediawiki.org needs some cleanups. We had some discussions about this theme few days ago.
Best
xqt
----- Original Nachricht ----
Von: Bináris <wikiposta(a)gmail.com>
An: Xqt <info(a)gno.de>
Datum: 02.06.2014 09:11
Betreff: Re: Re: [Pywikipedia-l] Deprecation of compat
> 2014-06-02 9:01 GMT+02:00 <info(a)gno.de>:
>
> > As I wroth with my last mail this behaviour is available since 3th june
> > 2012 using the pwb.py wrapper script.
> >
>
> That wasn't quite clear for me at once, but thnak you, this is a great
> news!
>
_______________________________________________
Pywikipedia-l mailing list
Pywikipedia-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikipedia-l
--
Amir
I propose to create a tracking bug for all features present in
AutoWikiBrowser but not in our 'core' (e.g. gerrit:118795
<https://gerrit.wikimedia.org/r/118795>).
Even if the user targets of the two projects are different nowadays, as
different are their tasks, I think PWB is going to be the ultimate
replacement to AWB, given its extreme extensibility and X-platformness
(actually, only a real GUI is missing).