Python/Job issues

List overview All Threads
Download

newer

older

Global contrib count

PGP support for ts-admins address

DeltaQuad

13 Apr 2011 13 Apr '11

5:54 p.m.

Hello Everyone, I am having some issues with Job Scheduling with python. I always get: deltaquad@willow:~$ qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -m e -j y -o $HOME/UAA.out $HOME/UAA/UAA.py Your job 314397 ("UAA.py") has been submitted UAA.out says: newtask: exec of /sge62/default/spool/yarrow/job_scripts/314397 failed: No such file or directory . My files are set for CHMOD 774 . They all exist . It always returns the error What am I doing wrong? -------------------------- DeltaQuad http://enwp.org/User_Talk:DeltaQuad

Attachments:

attachment.htm (text/html — 6.0 KB)

Show replies by date

Jeremy Baron

13 Apr 13 Apr

6 p.m.

What does this produce: qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -m e -j y -o $HOME/set.out set -Jeremy On Apr 13, 2011 11:54 AM, "DeltaQuad" <deltaquad(a)live.ca> wrote:

...

DeltaQuad

6:06 p.m.

Can’t say right now as my school blocks FTP/SSH to the toolserver, will let you know later today. -------------------------- DeltaQuad <http://enwp.org/User_Talk:DeltaQuad> http://enwp.org/User_Talk:DeltaQuad From: toolserver-l-bounces(a)lists.wikimedia.org [mailto:toolserver-l-bounces@lists.wikimedia.org] On Behalf Of Jeremy Baron Sent: April-13-11 12:01 PM To: toolserver-l(a)lists.wikimedia.org Subject: Re: [Toolserver-l] Python/Job issues What does this produce: qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -m e -j y -o $HOME/set.out set -Jeremy On Apr 13, 2011 11:54 AM, "DeltaQuad" <deltaquad(a)live.ca> wrote:

...

Jeremy Baron

6:14 p.m.

:-/ Maybe someone should set up a TS login server sshd to listen on port 443 aka https. (if it's not already claimed by a web service) -Jeremy On Apr 13, 2011 12:06 PM, "DeltaQuad" <deltaquad(a)live.ca> wrote:

...

Can’t say right now as my school blocks FTP/SSH to the toolserver, will

let you know later today.

...

-------------------------- DeltaQuad <http://enwp.org/User_Talk:DeltaQuad> http://enwp.org/User_Talk:DeltaQuad From: toolserver-l-bounces(a)lists.wikimedia.org [mailto:

toolserver-l-bounces(a)lists.wikimedia.org] On Behalf Of Jeremy Baron

...

Sent: April-13-11 12:01 PM To: toolserver-l(a)lists.wikimedia.org Subject: Re: [Toolserver-l] Python/Job issues What does this produce: qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -m e -j y -o $HOME/set.out set -Jeremy On Apr 13, 2011 11:54 AM, "DeltaQuad" <deltaquad(a)live.ca> wrote: > Hello Everyone, > > > > I am having some issues with Job Scheduling with python. I always get: > > > > deltaquad@willow:~$ qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -m e -j y -o > $HOME/UAA.out $HOME/UAA/UAA.py > > Your job 314397 ("UAA.py") has been submitted > > > > UAA.out says: > > newtask: exec of /sge62/default/spool/yarrow/job_scripts/314397 failed:

...

such file or directory . My files are set for CHMOD 774 . They all exist . It always returns the error What am I doing wrong? -------------------------- DeltaQuad http://enwp.org/User_Talk:DeltaQuad

Simon Walker

7:17 p.m.

On 13 April 2011 17:14, Jeremy Baron <jeremy(a)tuxmachine.com> wrote:

...

Maybe someone should set up a TS login server sshd to listen on port 443

aka https. (if it's not already claimed by a web service) An alternate SSH port would be trivial to set up, but 443 maybe isn't the best port, as this would prevent possible future access to the toolserver over https (unless it was changed)

...

On Apr 13, 2011 12:06 PM, "DeltaQuad" <deltaquad(a)live.ca> wrote: > Can’t say right now as my school blocks FTP/SSH to the toolserver, will

let you know later today. Heh, I can't say I blame them. SSH was always my way around the filtering at school, combined with portable firefox. (socks5 proxy using dynamic port forwarding if you want to know :P) Simon

Ilmari Karonen

11:13 p.m.

On 04/13/2011 08:17 PM, Simon Walker wrote:

...

On 13 April 2011 17:14, Jeremy Baron <jeremy(a)tuxmachine.com <mailto:jeremy@tuxmachine.com>> wrote:

Maybe someone should set up a TS login server sshd to listen on port

443 aka https. (if it's not already claimed by a web service) An alternate SSH port would be trivial to set up, but 443 maybe isn't the best port, as this would prevent possible future access to the toolserver over https (unless it was changed)

If it's only on a login server, I don't see any problem. Those servers shouldn't be serving web pages anyway. Not sure if there's really that much need for it, though.

...

On Apr 13, 2011 12:06 PM, "DeltaQuad" <deltaquad(a)live.ca

<mailto:deltaquad@live.ca>> wrote:

> Can’t say right now as my school blocks FTP/SSH to the toolserver,

will let you know later today. Heh, I can't say I blame them. SSH was always my way around the filtering at school, combined with portable firefox. (socks5 proxy using dynamic port forwarding if you want to know :P)

BTW, if you have access to an HTTP(S) proxy, you may be able to tunnel SSH over that. Just Google for "ssh over https". -- Ilmari Karonen

DeltaQuad

9:34 p.m.

It produces the command line error: Unable to read script file because of error: error opening set: No such file or directory Instead of the text file. -------------------------- DeltaQuad <http://enwp.org/User_Talk:DeltaQuad> http://enwp.org/User_Talk:DeltaQuad From: toolserver-l-bounces(a)lists.wikimedia.org [mailto:toolserver-l-bounces@lists.wikimedia.org] On Behalf Of Jeremy Baron Sent: April-13-11 12:14 PM To: toolserver-l(a)lists.wikimedia.org Subject: Re: [Toolserver-l] Python/Job issues :-/ Maybe someone should set up a TS login server sshd to listen on port 443 aka https. (if it's not already claimed by a web service) -Jeremy On Apr 13, 2011 12:06 PM, "DeltaQuad" <deltaquad(a)live.ca> wrote:

...

MZMcBride

14 Apr 14 Apr

7:52 a.m.

DeltaQuad wrote:

...

I am having some issues with Job Scheduling with python. I always get: deltaquad@willow:~$ qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -m e -j y -o $HOME/UAA.out $HOME/UAA/UAA.py Your job 314397 ("UAA.py") has been submitted UAA.out says: newtask: exec of /sge62/default/spool/yarrow/job_scripts/314397 failed: No such file or directory

As far as I can see, this is due to /home/deltaquad/UAA/UAA.py using Windows line endings (CRLF) instead of Unix line endings (LF). MZMcBride

Marcin Cieslak

12:02 p.m.

...

deltaquad@willow:~$ qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -m e -j y -o $HOME/UAA.out $HOME/UAA/UAA.py Your job 314397 ("UAA.py") has been submitted

Can you try this: qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -b y -m e -j y -o $HOME/UAA.out $HOME/UAA/UAA.py (I added the "-b y" option and it seems to work here). //Marcin

DeltaQuad

1:47 p.m.

...

You should read <https://wiki.toolserver.org/view/Mailing_list_etiquette>

when you get a chance. Sorry about that. Read it now.

...

As far as I can see, this is due to /home/deltaquad/UAA/UAA.py using

Windows line endings (CRLF) instead of Unix line endings (LF). I've ran this on my Linux Mint (with the change of the pywikipedia directory only) and it runs fine. (Not sure if that helps)

...

Can you try this:

...

qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -b y -m e -j y -o $HOME/UAA.out

$HOME/UAA/UAA.py

...

(I added the "-b y" option and it seems to work here).

I get this instead in the output file now: " newtask: exec of /home/deltaquad/UAA/UAA.py failed: No such file or directory" -------------------------- DeltaQuad http://enwp.org/User_Talk:DeltaQuad

Marcin Cieslak

2:50 p.m.

...

DeltaQuad <deltaquad(a)live.ca> wrote: You should read <https://wiki.toolserver.org/view/Mailing_list_etiquette>

when you get a chance. Sorry about that. Read it now.

As far as I can see, this is due to /home/deltaquad/UAA/UAA.py using

Windows line endings (CRLF) instead of Unix line endings (LF). I've ran this on my Linux Mint (with the change of the pywikipedia directory only) and it runs fine. (Not sure if that helps)

Can you try this:

qsub -l h_rt=0:05:00 -l s_rt=0:04:30 -b y -m e -j y -o $HOME/UAA.out

$HOME/UAA/UAA.py

(I added the "-b y" option and it seems to work here).

I get this instead in the output file now: " newtask: exec of /home/deltaquad/UAA/UAA.py failed: No such file or directory"

This is because your CRLF Windows line endings in your file. I copied your file to my $HOME and got the same problem. SGE is not exactly the same as running your script from shell. Besides, you don't need #! /usr/bin/python twice and if you use "-b y" your #$ declarations won't work and should be removed. //Marcin

MZMcBride

15 Apr 15 Apr

2:18 a.m.

Marcin Cieslak wrote:

...

This is because your CRLF Windows line endings in your file. I copied your file to my $HOME and got the same problem.

Right. I said this already. ;-)

...

SGE is not exactly the same as running your script from shell.

...

From a bit of testing, it seems like Python doesn't care if a script uses

CRLF line endings or LF line endings, but Unix's exec() does. When SGE calls exec() on the Python script with the CRLF line endings, it can't get past the first line and then dies with a mostly unhelpful error message. DeltaQuad: I tested this pretty extensively last night. Seriously, just fix your line endings and you should be good to go. :-) MZMcBride

4759

days inactive

4761

days old

toolserver-l@lists.wikimedia.org

Manage subscription

11 comments

6 participants

tags (0)

participants (6)

DeltaQuad
Ilmari Karonen
Jeremy Baron
Marcin Cieslak
MZMcBride
Simon Walker