<div dir="ltr">Hi all,<div><br></div><div>This is a combination of a Python 3 design choice (PEP 383 [1]) and T60786 [2]. What happens is the following:</div><div><br></div><div>1) The locale is set to a encoding that cannot decode certain bytes -- for example, ASCII, which can only decode bytes < 128.</div><div>2) Python is started with a command line parameter that contains a byte > 128 (\x80), for example, "ř' when UTF-8 encoded is represented by two bytes: \xc5\x99. Both of these are > \x80, and can therefore not be interpreted as ASCII</div><div>3) Python3 needs to somehow decode these bytes into a text string. But there is no valid way to do so! Instead of complaining loudly with a UnicodeDecodeError, Python3 embeds the bytes as 'fake characters' in the string -- as described in PEP 383.</div><div>\xc5\x59 is therefore now suddenly decoded as "'\udcc5\udc99". instead of "ř".</div><div>4) Pywikibot tries to encode these characters using utf-8, but they are fake characters, and the .encode step blows up.<br></div><div><br></div><div>A simple way to reproduce this is the following:</div><div><br></div><div>valhallasw@tools-bastion-03:~/ucm$ cat test.py<br></div><div><div>import sys</div><div>encoded = sys.argv[1].encode('utf-8')</div><div><br></div><div>valhallasw@tools-bastion-03:~/ucm$ LC_ALL=C python3 test.py řeklad<br></div><div>Traceback (most recent call last):</div><div> File "test.py", line 2, in <module></div><div> encoded = sys.argv[1].encode('utf-8')</div><div>UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc5' in position 0: surrogates not allowed</div></div><div><br></div><div>This should be fixed in future Python versions (likely 3.7), when PEP540 [3] is implemented.</div><div><br></div><div>As for the current situation, the simplest solution is to add 'export LC_ALL=en_US.UTF-8' to your script, before the 'python ...' line.</div><div><br></div><div>Best, </div><div>Merlijn</div><div><br></div><div>[1] <a href="https://www.python.org/dev/peps/pep-0383/">https://www.python.org/dev/peps/pep-0383/</a></div><div>[2] <a href="https://phabricator.wikimedia.org/T60784">https://phabricator.wikimedia.org/T60784</a></div><div>[3] <a href="https://www.python.org/dev/peps/pep-0540/">https://www.python.org/dev/peps/pep-0540/</a></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 16 June 2017 at 23:58, Bryan Davis <span dir="ltr"><<a href="mailto:bd808@wikimedia.org" target="_blank">bd808@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Fri, Jun 16, 2017 at 10:15 AM, Martin Urbanec<br>
<<a href="mailto:martin.urbanec@wikimedia.cz">martin.urbanec@wikimedia.cz</a>> wrote:<br>
> Hello,<br>
><br>
> I have a script which should add a template to articles which are created by<br>
> the ContentTranslation tool (the template has parameters which depends on<br>
> language and revision which were used as the source one; this is the reason<br>
> why I use separate script). It may be found at<br>
> <a href="https://github.com/urbanecm/addPrekladCT/blob/master/addmissing.py" rel="noreferrer" target="_blank">https://github.com/urbanecm/<wbr>addPrekladCT/blob/master/<wbr>addmissing.py</a>. The<br>
> script work perfectly on my local PC and on bastion host but I can't get it<br>
> work on the grid.<br>
><br>
> The script itself is run by python3 addmissing.py -always -file:pages.txt<br>
> -search:'-insource:/\{\{[Pp]<wbr>řeklad/' and require pages.txt file and<br>
> preklads.txt file at<br>
> <a href="https://tools.wmflabs.org/urbanecmbot/test/preklads.txt" rel="noreferrer" target="_blank">https://tools.wmflabs.org/<wbr>urbanecmbot/test/preklads.txt</a>. The first contains<br>
> pages that should be processed and act as the generator, the second one is<br>
> something like a database with exact templates which should be inserted.<br>
> Both files are as an example in the attachments.<br>
><br>
> When I try to run it at toollabs bastion, all works as it should. When I<br>
> send the script to grid, it do not work (see sample output below). Why? Can<br>
> somebody help me with it?<br>
><br>
> Thank you in advance,<br>
> Martin Urbanec / Urbanecm<br>
><br>
> ; Output<br>
><br>
> urbanecm@tools-bastion-02 ~/Documents/cswiki/<wbr>addPrekladCT<br>
> $ cat test.sh<br>
> python3 addmissing.py -always -file:pages.txt<br>
> -search:'-insource:/\{\{[Pp]<wbr>řeklad/'<br>
> urbanecm@tools-bastion-02 ~/Documents/cswiki/<wbr>addPrekladCT<br>
> $ jsub bash test.sh<br>
> Your job 6201363 ("bash") has been submitted<br>
> urbanecm@tools-bastion-02 ~/Documents/cswiki/<wbr>addPrekladCT<br>
> $ qstat<br>
> job-ID prior name user state submit/start at queue<br>
> slots ja-task-ID<br>
> ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-----------------------<br>
> 6201363 0.30000 bash urbanecm r 06/16/2017 18:14:42<br>
> task@tools-exec-1404.eqiad.wmf 1<br>
> urbanecm@tools-bastion-02 ~/Documents/cswiki/<wbr>addPrekladCT<br>
> $ ls ~/bash.*<br>
> /home/urbanecm/bash.err /home/urbanecm/bash.out<br>
> urbanecm@tools-bastion-02 ~/Documents/cswiki/<wbr>addPrekladCT<br>
> $ cat ~/bash.*<br>
> Traceback (most recent call last):<br>
> File "addmissing.py", line 223, in <module><br>
> main()<br>
> File "addmissing.py", line 183, in main<br>
> local_args = pywikibot.handle_args(args)<br>
> File "/shared/pywikipedia/core/<wbr>pywikibot/bot.py", line 954, in handle_args<br>
> writeToCommandLogFile()<br>
> File "/shared/pywikipedia/core/<wbr>pywikibot/bot.py", line 1128, in<br>
> writeToCommandLogFile<br>
> command_log_file.write(s + os.linesep)<br>
> File "/usr/lib/python3.4/codecs.py"<wbr>, line 711, in write<br>
> return self.writer.write(data)<br>
> File "/usr/lib/python3.4/codecs.py"<wbr>, line 368, in write<br>
> data, consumed = self.encode(object, self.errors)<br>
> UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc5' in<br>
> position 67: surrogates not allowed<br>
> CRITICAL: Closing network session.<br>
> <class 'UnicodeEncodeError'><br>
> urbanecm@tools-bastion-02 ~/Documents/cswiki/<wbr>addPrekladCT<br>
> $<br>
<br>
</div></div>Zhuyifei1999 saw your email and noted on irc that it looks to be a<br>
case of the known bug that I just retitled as "Shell LOCALE neither<br>
consistent nor sane across grid engine nodes"<br>
(<<a href="https://phabricator.wikimedia.org/T60784" rel="noreferrer" target="_blank">https://phabricator.<wbr>wikimedia.org/T60784</a>>). The current best work<br>
around that bug is to launch the job as a shell script that sets<br>
either LANG=C.UTF-8 or PYTHONIOENCODING=utf-8.<br>
<br>
If setting the job to run with the same locale you are using in your<br>
interactive tests does not work to fix the problem, you may also be<br>
hitting a deeper Python3 unicode issue related to surrogate codepoints<br>
(<<a href="https://bugs.python.org/issue12892" rel="noreferrer" target="_blank">https://bugs.python.org/<wbr>issue12892</a>>). This is hinted by the<br>
"position 67: surrogates not allowed" error message.<br>
<br>
I can actually reproduce your error message in an interactive python<br>
session on tools-dev from a starting state of LANG=en_US.UTF-8:<br>
<br>
$ python3<br>
Python 3.4.0 (default, Jun 19 2015, 14:20:21)<br>
[GCC 4.8.2] on linux<br>
Type "help", "copyright", "credits" or "license" for more information.<br>
>>> print('\udcc5')<br>
<span class=""> Traceback (most recent call last):<br>
</span> File "<stdin>", line 1, in <module><br>
<span class=""> UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc5' in<br>
</span>position 0: surrogates not allowed<br>
>>><br>
<br>
Explictly encoding using 'surrogateescape' does work:<br>
>>> print('\udcc5'.encode('utf-8', 'surrogateescape'))<br>
b'\xc5'<br>
<br>
It looks like the error could be dealt with in pywikibot by patching<br>
writeToCommandLogFile() to open the codec used for output with any<br>
value other than the default errors='strict'<br>
(<<a href="https://docs.python.org/3/library/codecs.html#error-handlers" rel="noreferrer" target="_blank">https://docs.python.org/3/<wbr>library/codecs.html#error-<wbr>handlers</a>>).<br>
<br>
$ python3<br>
Python 3.4.0 (default, Jun 19 2015, 14:20:21)<br>
[GCC 4.8.2] on linux<br>
Type "help", "copyright", "credits" or "license" for more information.<br>
>>> print('\udcc5'.encode('utf-8', 'ignore'))<br>
b''<br>
>>> print('\udcc5'.encode('utf-8', 'replace'))<br>
b'?'<br>
>>> print('\udcc5'.encode('utf-8', 'xmlcharrefreplace'))<br>
b'�'<br>
>>> print('\udcc5'.encode('utf-8', 'backslashreplace'))<br>
b'\\udcc5'<br>
>>> print('\udcc5'.encode('utf-8', 'surrogateescape'))<br>
b'\xc5'<br>
>>> print('\udcc5'.encode('utf-8', 'surrogatepass'))<br>
b'\xed\xb3\x85'<br>
>>><br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
Bryan<br>
--<br>
Bryan Davis Wikimedia Foundation <<a href="mailto:bd808@wikimedia.org">bd808@wikimedia.org</a>><br>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA<br>
irc: bd808 v:<a href="tel:415.839.6885%20x6855" value="+14158396885">415.839.6885 x6855</a><br>
</font></span><div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>_________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" rel="noreferrer" target="_blank">https://lists.wikimedia.org/<wbr>mailman/listinfo/labs-l</a><br>
</div></div></blockquote></div><br></div>