Hello,
I deleted one of my script, and try to recreate it. But I now have an encoding issue when I run it with SGE. I searched a solution during all this evening and can't find any idea to solve it. I hope one of you will be able to help me.
I created the file /home/hercule/batch/LienFr.sh with this content :
#-*- coding: utf-8 -*- #!bin/bash #! /usr/bin/python #$ -l h_rt=1:30:00 #$ -l virtual_free=500M #$ -l arch='*' #$ -m as #$ -o /home/hercule/logs/LienFr.log #$ -j y #$ -wd /home/hercule/ python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à la création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
When I simply run /home/hercule/batch/LienFr.sh in the terminal the result is correct : http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&am...
But when I use the command line qcronsub -N LienFr /home/hercule/batch/LienFr.sh
I have a problem with the summary :
http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&am...
Do you know how to solve this ?
Regards
Hercule
On 23/05/13 00:38, Hercule Hercule wrote:
Hello,
I deleted one of my script, and try to recreate it. But I now have an encoding issue when I run it with SGE. I searched a solution during all this evening and can't find any idea to solve it. I hope one of you will be able to help me.
I created the file /home/hercule/batch/LienFr.sh with this content :
#-*- coding: utf-8 -*- #!bin/bash #! /usr/bin/python
These shebangs are crazy. You would want just "#!/bin/bash" as the first line.
#$ -l h_rt=1:30:00 #$ -l virtual_free=500M #$ -l arch='*' #$ -m as #$ -o /home/hercule/logs/LienFr.log #$ -j y #$ -wd /home/hercule/ python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à la création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
When I simply run /home/hercule/batch/LienFr.sh in the terminal the result is correct : http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&am...
But when I use the command line qcronsub -N LienFr /home/hercule/batch/LienFr.sh
I have a problem with the summary :
http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&am...
Do you know how to solve this ?
Regards
Hercule
Seems the string was converted to utf-8 twice. What did you use to edit the file?
Try converting it with iconv: iconv -f utf-8 -t iso-8859-1 < LienFr.sh > LienFr2.sh
Hello,
If I don't have the three first lines, the log give
newtask: exec of /var/sge/spool/execd/wolfsbane/job_scripts/2016385 failed: No such file or directory
When making the iconv I have no difference at all. I tried using LienFr and LienFr2 with the same result.
Regards
Hercule
2013/5/23 Platonides platonides@gmail.com
On 23/05/13 00:38, Hercule Hercule wrote:
Hello,
I deleted one of my script, and try to recreate it. But I now have an encoding issue when I run it with SGE. I searched a solution during all this evening and can't find any idea to solve it. I hope one of you will be able to help me.
I created the file /home/hercule/batch/LienFr.sh with this content :
#-*- coding: utf-8 -*- #!bin/bash #! /usr/bin/python
These shebangs are crazy. You would want just "#!/bin/bash" as the first line.
#$ -l h_rt=1:30:00 #$ -l virtual_free=500M #$ -l arch='*' #$ -m as #$ -o /home/hercule/logs/LienFr.log #$ -j y #$ -wd /home/hercule/ python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à la création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
When I simply run /home/hercule/batch/LienFr.sh in the terminal the result is correct :
http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&am...
But when I use the command line qcronsub -N LienFr /home/hercule/batch/LienFr.sh
I have a problem with the summary :
http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&am...
Do you know how to solve this ?
Regards
Hercule
Seems the string was converted to utf-8 twice. What did you use to edit the file?
Try converting it with iconv: iconv -f utf-8 -t iso-8859-1 < LienFr.sh > LienFr2.sh
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Hi Hercule,
On 23 May 2013 00:38, Hercule Hercule hercule.wikipedia@gmail.com wrote:
python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à la création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
If you have saved the file in the utf-8 encoding (which the edit summary suggests), try using (on one line)
LANG=en_GB.UTF-8 python /home/hercule/template.py ...
or (on two lines) export LANG=en_GB.UTF-8 python /home/hercule/template.py ...
Basically, you have to tell pywikipedia which encoding you have used for the shell arguments - in your case UTF-8. Otherwise it will use latin-1 as default, which results in the mojibake you saw.
Merlijn
General chat • Re: Navigating Multiboot GRUB2 menu entries successfully. http://forum.porteus.org/viewtopic.php?t=2195&p=15042#p15042 On May 23, 2013 3:12 AM, "Merlijn van Deen" valhallasw@arctus.nl wrote:
Hi Hercule,
On 23 May 2013 00:38, Hercule Hercule hercule.wikipedia@gmail.com wrote:
python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à
la
création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
If you have saved the file in the utf-8 encoding (which the edit summary suggests), try using (on one line)
LANG=en_GB.UTF-8 python /home/hercule/template.py ...
or (on two lines) export LANG=en_GB.UTF-8 python /home/hercule/template.py ...
Basically, you have to tell pywikipedia which encoding you have used for the shell arguments - in your case UTF-8. Otherwise it will use latin-1 as default, which results in the mojibake you saw.
Merlijn
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
I already tried to add
export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
What I don't understand is that the problem only comes with SGE submit. And the command 'locale' in the shell shows en_US.UTF-8 for every property
I will try LANG=fr_FR.UTF-8 python /home/hercule/template.py this evening
Le 23 mai 2013 09:16, Merlijn van Deen valhallasw@arctus.nl a écrit :
Hi Hercule,
On 23 May 2013 00:38, Hercule Hercule hercule.wikipedia@gmail.com wrote:
python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à
la
création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
If you have saved the file in the utf-8 encoding (which the edit summary suggests), try using (on one line)
LANG=en_GB.UTF-8 python /home/hercule/template.py ...
or (on two lines) export LANG=en_GB.UTF-8 python /home/hercule/template.py ...
Basically, you have to tell pywikipedia which encoding you have used for the shell arguments - in your case UTF-8. Otherwise it will use latin-1 as default, which results in the mojibake you saw.
Merlijn
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
I just checked what locale returns under SGE, and this is en_US.UTF-8 - so setting locale should not change - so it's not really clear to me why this error would occur.
In any case, you can try setting
console_encoding = 'utf-8'
in your user-config.py. That should take care of the problem, irrespective of locale setting.
Merlijn
On 23 May 2013 14:40, Hercule Hercule hercule.wikipedia@gmail.com wrote:
I already tried to add
export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
What I don't understand is that the problem only comes with SGE submit. And the command 'locale' in the shell shows en_US.UTF-8 for every property
I will try LANG=fr_FR.UTF-8 python /home/hercule/template.py this evening
Le 23 mai 2013 09:16, Merlijn van Deen valhallasw@arctus.nl a écrit :
Hi Hercule,
On 23 May 2013 00:38, Hercule Hercule hercule.wikipedia@gmail.com wrote:
python /home/hercule/template.py "Lien" "Lien/Conversion automatique" -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] : transformation de liens avec le modèle {{Lien}} en lien interne, suite à la création de l'article correspondant" -assubst -pt:0 -always mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
If you have saved the file in the utf-8 encoding (which the edit summary suggests), try using (on one line)
LANG=en_GB.UTF-8 python /home/hercule/template.py ...
or (on two lines) export LANG=en_GB.UTF-8 python /home/hercule/template.py ...
Basically, you have to tell pywikipedia which encoding you have used for the shell arguments - in your case UTF-8. Otherwise it will use latin-1 as default, which results in the mojibake you saw.
Merlijn
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
On 05/23/2013 02:40 PM, Hercule Hercule wrote:
I already tried to add
export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
The syntax "export A=B" is from bash, a common login shell. However, most scripts are traditionally written in #!/bin/sh where the syntax requires "export A" and "A=B" to be written as two separate statements.
(Knowledge from the 1980s.)
I added export LANG LANG=fr_FR.UTF-8 in the script, but it still make incorrect comments.
I add enclosed the sh file, if it can help
2013/5/23 Lars Aronsson lars@aronsson.se
On 05/23/2013 02:40 PM, Hercule Hercule wrote:
I already tried to add
export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
The syntax "export A=B" is from bash, a common login shell. However, most scripts are traditionally written in #!/bin/sh where the syntax requires "export A" and "A=B" to be written as two separate statements.
(Knowledge from the 1980s.)
-- Lars Aronsson (lars@aronsson.se) Aronsson Datateknik - http://aronsson.se
______________________________**_________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette
The addition of console_encoding = 'utf-8' in my user-config.py solved the issue. Thanks everybody for your help
2013/5/23 Hercule Hercule hercule.wikipedia@gmail.com
I added export LANG LANG=fr_FR.UTF-8 in the script, but it still make incorrect comments.
I add enclosed the sh file, if it can help
2013/5/23 Lars Aronsson lars@aronsson.se
On 05/23/2013 02:40 PM, Hercule Hercule wrote:
I already tried to add
export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
The syntax "export A=B" is from bash, a common login shell. However, most scripts are traditionally written in #!/bin/sh where the syntax requires "export A" and "A=B" to be written as two separate statements.
(Knowledge from the 1980s.)
-- Lars Aronsson (lars@aronsson.se) Aronsson Datateknik - http://aronsson.se
______________________________**_________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette
Your welcome if I had any part in helping you out. On May 23, 2013 2:31 PM, "Hercule Hercule" hercule.wikipedia@gmail.com wrote:
The addition of console_encoding = 'utf-8' in my user-config.py solved the issue. Thanks everybody for your help
2013/5/23 Hercule Hercule hercule.wikipedia@gmail.com
I added export LANG LANG=fr_FR.UTF-8 in the script, but it still make incorrect comments.
I add enclosed the sh file, if it can help
2013/5/23 Lars Aronsson lars@aronsson.se
On 05/23/2013 02:40 PM, Hercule Hercule wrote:
I already tried to add
export LANG=fr_FR.UTF-8 but I then have an alert : LANG=fr_FR.UTF-8 is not an identifier LANG=fr_FR.UTF-8 at the top of the shell don't change anything.
The syntax "export A=B" is from bash, a common login shell. However, most scripts are traditionally written in #!/bin/sh where the syntax requires "export A" and "A=B" to be written as two separate statements.
(Knowledge from the 1980s.)
-- Lars Aronsson (lars@aronsson.se) Aronsson Datateknik - http://aronsson.se
______________________________**_________________ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.**orgToolserver-l@lists.wikimedia.org ) https://lists.wikimedia.org/**mailman/listinfo/toolserver-lhttps://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/** view/Mailing_list_etiquettehttps://wiki.toolserver.org/view/Mailing_list_etiquette
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
toolserver-l@lists.wikimedia.org