https://bugzilla.wikimedia.org/show_bug.cgi?id=72009
Bug ID: 72009 Summary: FC20 ships httplib2 0.7.7 causing endless SSL cert fail loop on python3 Product: Pywikibot Version: core (2.0) Hardware: PC OS: Linux Status: NEW Severity: major Priority: Unprioritized Component: network Assignee: Pywikipedia-bugs@lists.wikimedia.org Reporter: jayvdb@gmail.com Web browser: --- Mobile Platform: ---
IIRC, there is a problem with certificates in httplib2 pre-0.9 (bug 65189). FC20 ships 0.7.7 on python2 and python3
$ rpm -qi python-httplib2 Name : python-httplib2 Version : 0.7.7 Release : 3.fc20 ... $ rpm -qi python3-httplib2 Name : python3-httplib2 Version : 0.7.7 Release : 3.fc20 ...
On py2, the 0.7.7 package works correctly. On py3, the following occurs running site_tests
On py2, all tests pass using 0.7.7. On python3, tests fail badly (using en.wp as default site).
$ python3 pwb.py tests/site_tests.py .../pywikibot/family.py:879: DeprecationWarning: imp.load_source() is deprecated; use importlib.machinery.SourceFileLoader(name, pathname).load_module() instead myfamily = imp.load_source(fam, config.family_files[fam]) ERROR: Traceback (most recent call last): File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 992, in _conn_request response = conn.getresponse() File "/usr/lib64/python3.3/http/client.py", line 1135, in getresponse raise ResponseNotReady(self.__state) http.client.ResponseNotReady: Idle
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File ".../pywikibot/data/api.py", line 517, in submit headers=headers, body=body) File ".../pywikibot/tools.py", line 367, in wrapper return method(*__args, **__kw) File ".../pywikibot/comms/http.py", line 258, in request raise request.data File ".../pywikibot/comms/threadedhttp.py", line 219, in request max_redirects, connection_type File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1272, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1026, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 999, in _conn_request conn.connect() File "/usr/lib64/python3.3/http/client.py", line 1202, in connect server_hostname=server_hostname) File "/usr/lib64/python3.3/ssl.py", line 245, in wrap_socket _context=self) File "/usr/lib64/python3.3/ssl.py", line 345, in __init__ raise x File "/usr/lib64/python3.3/ssl.py", line 341, in __init__ self.do_handshake() File "/usr/lib64/python3.3/ssl.py", line 548, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:547)
WARNING: Waiting 5 seconds before retrying. ERROR: Traceback (most recent call last): File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 992, in _conn_request response = conn.getresponse() File "/usr/lib64/python3.3/http/client.py", line 1135, in getresponse raise ResponseNotReady(self.__state) http.client.ResponseNotReady: Idle
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File ".../pywikibot/data/api.py", line 517, in submit headers=headers, body=body) File ".../pywikibot/tools.py", line 367, in wrapper return method(*__args, **__kw) File ".../pywikibot/comms/http.py", line 258, in request raise request.data File ".../pywikibot/comms/threadedhttp.py", line 219, in request max_redirects, connection_type File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1272, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1026, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 999, in _conn_request conn.connect() File "/usr/lib64/python3.3/http/client.py", line 1202, in connect server_hostname=server_hostname) File "/usr/lib64/python3.3/ssl.py", line 245, in wrap_socket _context=self) File "/usr/lib64/python3.3/ssl.py", line 345, in __init__ raise x File "/usr/lib64/python3.3/ssl.py", line 341, in __init__ self.do_handshake() File "/usr/lib64/python3.3/ssl.py", line 548, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:547)
WARNING: Waiting 10 seconds before retrying. ^CTraceback (most recent call last): File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 992, in _conn_request response = conn.getresponse() File "/usr/lib64/python3.3/http/client.py", line 1135, in getresponse raise ResponseNotReady(self.__state) http.client.ResponseNotReady: Idle
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File ".../pywikibot/data/api.py", line 517, in submit headers=headers, body=body) File ".../pywikibot/tools.py", line 367, in wrapper return method(*__args, **__kw) File ".../pywikibot/comms/http.py", line 258, in request raise request.data File ".../pywikibot/comms/threadedhttp.py", line 219, in request max_redirects, connection_type File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1272, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 1026, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/usr/lib/python3.3/site-packages/httplib2/__init__.py", line 999, in _conn_request conn.connect() File "/usr/lib64/python3.3/http/client.py", line 1202, in connect server_hostname=server_hostname) File "/usr/lib64/python3.3/ssl.py", line 245, in wrap_socket _context=self) File "/usr/lib64/python3.3/ssl.py", line 345, in __init__ raise x File "/usr/lib64/python3.3/ssl.py", line 341, in __init__ self.do_handshake() File "/usr/lib64/python3.3/ssl.py", line 548, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:547)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "pwb.py", line 164, in <module> run_python_file(fn, argv, argvu) File "pwb.py", line 69, in run_python_file exec(compile(source, filename, "exec"), main_mod.__dict__) File "tests/site_tests.py", line 1481, in <module> unittest.main() File "/usr/lib64/python3.3/unittest/main.py", line 125, in __init__ self.runTests() File "/usr/lib64/python3.3/unittest/main.py", line 265, in runTests self.result = testRunner.run(self.test) File "/usr/lib64/python3.3/unittest/runner.py", line 168, in run test(result) File "/usr/lib64/python3.3/unittest/suite.py", line 67, in __call__ return self.run(*args, **kwds) File "/usr/lib64/python3.3/unittest/suite.py", line 105, in run test(result) File "/usr/lib64/python3.3/unittest/suite.py", line 67, in __call__ return self.run(*args, **kwds) File "/usr/lib64/python3.3/unittest/suite.py", line 105, in run test(result) File "/usr/lib64/python3.3/unittest/case.py", line 529, in __call__ return self.run(*args, **kwds) File "/usr/lib64/python3.3/unittest/case.py", line 477, in run self._executeTestPart(testMethod, outcome, isTest=True) File "/usr/lib64/python3.3/unittest/case.py", line 422, in _executeTestPart function() File "tests/site_tests.py", line 1065, in testRandompages rn = list(mysite.randompages(total=10)) File ".../pywikibot/site.py", line 3574, in randompages g_content=content) File ".../pywikibot/site.py", line 1357, in _generator gen = gen_class(type_arg, site=self, **args) File ".../pywikibot/data/api.py", line 1100, in __init__ QueryGenerator.__init__(self, generator=generator, **kwargs) File ".../pywikibot/data/api.py", line 819, in __init__ self.update_limit() # sets self.prefix File ".../pywikibot/data/api.py", line 914, in update_limit for param in self._modules[mod].get("parameters", []): File ".../pywikibot/data/api.py", line 865, in _modules data = paramreq.submit() File ".../pywikibot/data/api.py", line 764, in submit self._data = super(CachedRequest, self).submit() File ".../pywikibot/data/api.py", line 535, in submit self.wait() File ".../pywikibot/data/api.py", line 651, in wait time.sleep(self.retry_wait) KeyboardInterrupt <class 'KeyboardInterrupt'> CRITICAL: Waiting for 1 network thread(s) to finish. Press ctrl-c to abort
https://bugzilla.wikimedia.org/show_bug.cgi?id=72009
Andre Klapper aklapper@wikimedia.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Summary|FC20 ships httplib2 0.7.7 |Fedora 20 ships httplib2 |causing endless SSL cert |0.7.7 causing endless SSL |fail loop on python3 |cert fail loop on python3
https://bugzilla.wikimedia.org/show_bug.cgi?id=72009
John Mark Vandenberg jayvdb@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://bugzilla.wikimedia. | |org/show_bug.cgi?id=72236
--- Comment #1 from John Mark Vandenberg jayvdb@gmail.com --- In addition, setting Family.ignore_certificate_error does not work in py3 (bug 72236 - upstream)
https://bugzilla.wikimedia.org/show_bug.cgi?id=72009
John Mark Vandenberg jayvdb@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Depends on| |72247
https://bugzilla.wikimedia.org/show_bug.cgi?id=72009
John Mark Vandenberg jayvdb@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Keywords| |upstream See Also| |https://bugzilla.redhat.com | |/show_bug.cgi?id=1154641 Summary|Fedora 20 ships httplib2 |Fedora 20 ships python3 |0.7.7 causing endless SSL |httplib2 0.7.7 with |cert fail loop on python3 |unpatched SSL cert file Severity|major |normal
--- Comment #2 from John Mark Vandenberg jayvdb@gmail.com --- The reason for the problem with py3 is Fedora patches httplib2-0.7.7 to use the system certs, but only for py2. Therefore python3-httplib2-0.7.7 is using the certs included in the distributed 0.7.7 tarball, which is mostly junk data nowadays.
fwiw, it looks like FC21 is likely to be based on py34 and include httplib2-0.9
Our typical workarounds for httplib2 work well; either install 0.9 using pip, or recursively clone the pywikibot repo, so it can use 'externals/httplib2' - both tested and work correctly.
https://bugzilla.wikimedia.org/show_bug.cgi?id=72009 Bug 72009 depends on bug 72247, which changed state.
Bug 72247 Summary: http module in py3 doesnt detect and treat SSL certificate errors correctly https://bugzilla.wikimedia.org/show_bug.cgi?id=72247
What |Removed |Added ---------------------------------------------------------------------------- Status|PATCH_TO_REVIEW |RESOLVED Resolution|--- |FIXED
pywikipedia-bugs@lists.wikimedia.org