Encoding issue using SGE

List overview All Threads
Download

newer

older

Status of the toolserver

Patricia Pintilie...

Hercule Hercule

23 May 2013 23 May '13

1:38 a.m.

Hello, I deleted one of my script, and try to recreate it. But I now have an encoding issue when I run it with SGE. I searched a solution during all this evening and can't find any idea to solve it. I hope one of you will be able to help me. I created the file /home/hercule/batch/LienFr.sh with this content : #-*- coding: utf-8 -*- #!bin/bash #! /usr/bin/python #$ -l h_rt=1:30:00 #$ -l virtual_free=500M #$ -l arch='*' #$ -m as #$ -o /home/hercule/logs/LienFr.log #$ -j y #$ -wd /home/hercule/ python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à la création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log When I simply run /home/hercule/batch/LienFr.sh in the terminal the result is correct : http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&a… But when I use the command line qcronsub -N LienFr /home/hercule/batch/LienFr.sh I have a problem with the summary : http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&a… Do you know how to solve this ? Regards Hercule

Attachments:

attachment.htm (text/html — 2.0 KB)

Show replies by date

Platonides

23 May 23 May

2:27 a.m.

On 23/05/13 00:38, Hercule Hercule wrote:

...

These shebangs are crazy. You would want just "#!/bin/bash" as the first line.

...

#$ -l h_rt=1:30:00 #$ -l virtual_free=500M #$ -l arch='*' #$ -m as #$ -o /home/hercule/logs/LienFr.log #$ -j y #$ -wd /home/hercule/ python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à la création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log When I simply run /home/hercule/batch/LienFr.sh in the terminal the result is correct : http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&a… But when I use the command line qcronsub -N LienFr /home/hercule/batch/LienFr.sh I have a problem with the summary : http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&a… Do you know how to solve this ? Regards Hercule

Seems the string was converted to utf-8 twice. What did you use to edit the file? Try converting it with iconv: iconv -f utf-8 -t iso-8859-1 < LienFr.sh > LienFr2.sh

Hercule Hercule

3:01 a.m.

Hello, If I don't have the three first lines, the log give newtask: exec of /var/sge/spool/execd/wolfsbane/job_scripts/2016385 failed: No such file or directory When making the iconv I have no difference at all. I tried using LienFr and LienFr2 with the same result. Regards Hercule 2013/5/23 Platonides <platonides(a)gmail.com>

...

On 23/05/13 00:38, Hercule Hercule wrote:

These shebangs are crazy. You would want just "#!/bin/bash" as the first line.

http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&a…

But when I use the command line qcronsub -N LienFr /home/hercule/batch/LienFr.sh I have a problem with the summary :

http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&a…

Do you know how to solve this ? Regards Hercule

Seems the string was converted to utf-8 twice. What did you use to edit the file? Try converting it with iconv: iconv -f utf-8 -t iso-8859-1 < LienFr.sh > LienFr2.sh _______________________________________________ Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

Merlijn van Deen

10:16 a.m.

Hi Hercule, On 23 May 2013 00:38, Hercule Hercule <hercule.wikipedia(a)gmail.com> wrote:

...

python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à la création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log

Patricia Pintilie

2:30 p.m.

General chat • Re: Navigating Multiboot GRUB2 menu entries successfully. http://forum.porteus.org/viewtopic.php?t=2195&p=15042#p15042 On May 23, 2013 3:12 AM, "Merlijn van Deen" <valhallasw(a)arctus.nl> wrote:

...

Hi Hercule, On 23 May 2013 00:38, Hercule Hercule <hercule.wikipedia(a)gmail.com> wrote:

création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log

If you have saved the file in the utf-8 encoding (which the edit summary suggests), try using (on one line) LANG=en_GB.UTF-8 python /home/hercule/template.py ... or (on two lines) export LANG=en_GB.UTF-8 python /home/hercule/template.py ... Basically, you have to tell pywikipedia which encoding you have used for the shell arguments - in your case UTF-8. Otherwise it will use latin-1 as default, which results in the mojibake you saw. Merlijn _______________________________________________ Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

Hercule Hercule

3:40 p.m.

I already tried to add export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything. What I don't understand is that the problem only comes with SGE submit. And the command 'locale' in the shell shows en_US.UTF-8 for every property I will try LANG=fr_FR.UTF-8 python /home/hercule/template.py this evening Le 23 mai 2013 09:16, Merlijn van Deen <valhallasw(a)arctus.nl> a écrit :

...

Hi Hercule, On 23 May 2013 00:38, Hercule Hercule <hercule.wikipedia(a)gmail.com> wrote:

création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log

Merlijn van Deen

4:14 p.m.

I just checked what locale returns under SGE, and this is en_US.UTF-8 - so setting locale should not change - so it's not really clear to me why this error would occur. In any case, you can try setting console_encoding = 'utf-8' in your user-config.py. That should take care of the problem, irrespective of locale setting. Merlijn On 23 May 2013 14:40, Hercule Hercule <hercule.wikipedia(a)gmail.com> wrote:

...

Hi Hercule, On 23 May 2013 00:38, Hercule Hercule <hercule.wikipedia(a)gmail.com> wrote:

_______________________________________________ Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

Lars Aronsson

5:14 p.m.

On 05/23/2013 02:40 PM, Hercule Hercule wrote:

...

I already tried to add export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.

Hercule Hercule

9:31 p.m.

I added export LANG LANG=fr_FR.UTF-8 in the script, but it still make incorrect comments. I add enclosed the sh file, if it can help 2013/5/23 Lars Aronsson <lars(a)aronsson.se>

...

On 05/23/2013 02:40 PM, Hercule Hercule wrote:

I already tried to add export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.

The syntax "export A=B" is from bash, a common login shell. However, most scripts are traditionally written in #!/bin/sh where the syntax requires "export A" and "A=B" to be written as two separate statements. (Knowledge from the 1980s.) -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se ______________________________**_________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**org<Toolserver-l@lists.wikimedia.org> ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-l<https://list… Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquette<https://wiki.toolserver.org/view/Mailing_lis…

Hercule Hercule

10:30 p.m.

The addition of console_encoding = 'utf-8' in my user-config.py solved the issue. Thanks everybody for your help 2013/5/23 Hercule Hercule <hercule.wikipedia(a)gmail.com>

...

I added export LANG LANG=fr_FR.UTF-8 in the script, but it still make incorrect comments. I add enclosed the sh file, if it can help 2013/5/23 Lars Aronsson <lars(a)aronsson.se>

On 05/23/2013 02:40 PM, Hercule Hercule wrote:

I already tried to add export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.

Patricia Pintilie

24 May 24 May

12:19 a.m.

Your welcome if I had any part in helping you out. On May 23, 2013 2:31 PM, "Hercule Hercule" <hercule.wikipedia(a)gmail.com> wrote:

...

The addition of console_encoding = 'utf-8' in my user-config.py solved the issue. Thanks everybody for your help 2013/5/23 Hercule Hercule <hercule.wikipedia(a)gmail.com>

I added export LANG LANG=fr_FR.UTF-8 in the script, but it still make incorrect comments. I add enclosed the sh file, if it can help 2013/5/23 Lars Aronsson <lars(a)aronsson.se>

On 05/23/2013 02:40 PM, Hercule Hercule wrote:

I already tried to add export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.

4004

days inactive

4005

days old

toolserver-l@lists.wikimedia.org

Manage subscription

10 comments

5 participants

tags (0)

participants (5)

Hercule Hercule
Lars Aronsson
Merlijn van Deen
Patricia Pintilie
Platonides