[ pywikipediabot-Bugs-3160376 ] Non-ascii cookies not supported - Pywikipedia-bugs

25 Mar 2012


      Bugs item #3160376, was opened at 2011-01-17 16:08
Message generated for change (Comment added) made by nobody
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=3160376...
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: network
Group: None
Status: Open
Resolution: None
Priority: 7
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Non-ascii cookies not supported
Initial Comment:
Not fully clear whether this is a pwb or a python bug.
Reported by: Creeper == http://callofduty.wikia.com/wiki/User:Ikin
Wiki: callofduty.wikia.com, autogenerated family file
Location: Somewhere where wikia puts an å in the cookie.
Initial output:
C:\pywikipedia>redirect.py double
unicode test: triggers problem #3081100
Retrieving special page...
'ascii' codec can't decode byte 0xe5 in position 282: ordinal not in range(128)
WARNING: Could not open 'http://callofduty.wikia.com/api.php?action=query&format
=json&meta=userinfo&uiprop=blockinfo'. Maybe the server or
 your connection is down. Retrying in 1 minutes...
After removing two catch-alls:
00:10 < valhalla1w> open wikipedia.py, search for output(u'%s' %e)
00:10 < valhalla1w> change that to 'raise'
00:11 < valhalla1w> replace output(...) with raise
00:11 < valhalla1w> then search again and replace again
00:12 < valhalla1w> (there are two hits)
and
00:41 < valhalla1w> open query.py, search for Error downloading data
00:41 < valhalla1w> change that line to 'raise', too, and try again
and some pdb magic:
C:\pywikipedia> c:\python26\python -m pdb redirect.py double
...
c:\pywikipedia\redirect.py(53)<module>()
-> """
(Pdb) c
unicode test: triggers problem #3081100
Retrieving special page...
Traceback (most recent call last):
  File "c:\python26\lib\pdb.py", line 1296, in main
    pdb._runscript(mainpyfile)
  File "c:\python26\lib\pdb.py", line 1215, in _runscript
    self.run(statement)
  File "c:\python26\lib\bdb.py", line 372, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "redirect.py", line 911, in <module>
    main()
  File "redirect.py", line 907, in main
    bot.run()
  File "redirect.py", line 823, in run
    self.fix_double_redirects()
  File "redirect.py", line 650, in fix_double_redirects
    for redir_name in self.generator.retrieve_double_redirects():
  File "redirect.py", line 463, in retrieve_double_redirects
    maintenance_txt = self.site.getUrl(path)
  File "wikipedia.py", line 5469, in getUrl
    self._getUserDataOld(text, sysop = sysop)
  File "wikipedia.py", line 5597, in _getUserDataOld
    blocked = self._getBlock(sysop = sysop)
  File "wikipedia.py", line 4910, in _getBlock
    data = query.GetData(params, self)
  File "query.py", line 127, in GetData
    jsontext = site.getUrl( path, retry=True, sysop=sysop, data=data)
  File "wikipedia.py", line 5365, in getUrl
    f = MyURLopener.open(request)
  File "c:\python26\lib\urllib2.py", line 391, in open
    response = self._open(req, data)
  File "c:\python26\lib\urllib2.py", line 409, in _open
    '_open', req)
  File "c:\python26\lib\urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "c:\python26\lib\urllib2.py", line 1170, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "c:\python26\lib\urllib2.py", line 1142, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "c:\python26\lib\httplib.py", line 914, in request
    self._send_request(method, url, body, headers)
  File "c:\python26\lib\httplib.py", line 951, in _send_request
    self.endheaders()
  File "c:\python26\lib\httplib.py", line 908, in endheaders
    self._send_output()
  File "c:\python26\lib\httplib.py", line 778, in _send_output
    msg = "\r\n".join(self._buffer)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 291: ordina
l not in range(128)
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
...
c:\python26\lib\httplib.py(778)_send_output()
-> msg = "\r\n".join(self._buffer)
(Pdb)
(Pdb) self._buffer
[u'GET /api.php?action=query&format=json&meta=userinfo&uiprop=blockinfo HTTP/1.1
', u'Host: callofduty.wikia.com', 'User-Agent: PythonWikipediaBot/1.0', 'Cookie:
 cookieprefix=None; wikicitiesUserName=xxxt; loadtime=S1295308051.857639313,VS0
,AS69,MD14,DD102,AE538,VE753; wikicitiesToken=7xxxxxxx;
wikicities_session=xxxxx; varnish-stat=/server/LON/va
rnish-l8-LON/SESSION/; Geo = {"city":"Kabelv\xe5g","country":"NO","continent":"E
U"}; wikicitiesUserID=3116284; ', 'Connection: close', 'Accept-Encoding: gzip',
'', '']
Testing this directly confirms the problem:
...
...
...
buffer = [u'GET /api.php?action=query&format=json&meta=userinfo&uiprop=blockinfo HTTP/1.1', u'Host: callofduty.wikia.com', 'User-Agent: PythonWikipediaBot/1.0', 'Cookie: cookieprefix=None; wikicitiesUserName=xxxx; loadtime=S1295308051.857639313,VS0,AS69,MD14,DD102,AE538,VE753; wikicitiesToken=xxxx;wikicities_session=xxx; varnish-stat=/server/LON/varnish-l8-LON/SESSION/; Geo = {"city":"Kabelv\xe5g","country":"NO","continent":"EU"}; wikicitiesUserID=xxx; ', 'Connection: close', 'Accept-Encoding: gzip','', '']
"\r\n".join(buffer)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 290: ordinal not in range(128)
As a temporary solution, I suggested editing c:\python26\lib\httplib.py and replacing line
msg = "\r\n".join(self._buffer)
with
"\r\n".join([x.encode('utf-8') if isinstance(x, unicode) else x for x in self._buffer])
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2012-03-24 10:43
Message:
5PAM5y  <a href="http://ojpxkixdserx.com/">ojpxkixdserx</a>,
[url=http://rjxyvafeernp.com/%5Drjxyvafeernp%5B/url],
[link=http://jbfukiptqcvk.com/%5Djbfukiptqcvk%5B/link],
http://tfnmhhojnlev.com/
----------------------------------------------------------------------
Comment By: Merlijn S. van Deen (valhallasw)
Date: 2011-01-17 16:09
Message:
Correction: I submitted this.
----------------------------------------------------------------------
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=603138&aid=3160376...