In article <slrnimkfjp.fte.saper(a)saper.info>fo>, Marcin Cieslak <saper(a)saper.info> wrote: You should first run cronsub from the command-line with the same arguments. If that still doesn't work, post the entire command line and any output from it. Once you have it working from the command-line then it should be fine fromc ron. cronsub makes it easier to submit longrun jobs, automatically sets the default output file, runs the script from $HOME and prevents duplicate jobs being submitted. Otherwise it's identical to qsub. - river.

On Sun, Feb 27, 2011 at 1:10 PM, River Tarnell <r.tarnell(a)ieee.org> wrote: OK, so I gave up on starting my perl script using cronsub, which seems to work only if I give the perl script a parameter (WTF?), and instead altered it to be run manually, generating a shell script that could then start via cronsub. Script ( /home/magnus/cronjobs/most_wanted_wrapper.sh ) : #!/bin/ksh qsub -N mw_dewiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.dewiki_p.out -l sqlprocs-s5=1 /home/magnus/cronjobs/update_most_wanted.pl dewiki_p qsub -N mw_enwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.enwiki_p.out -l sqlprocs-s1=1 /home/magnus/cronjobs/update_most_wanted.pl enwiki_p qsub -N mw_frwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.frwiki_p.out -l sqlprocs-s6=1 /home/magnus/cronjobs/update_most_wanted.pl frwiki_p qsub -N mw_plwiki_p -j Y -o /home/magnus/cronjobs/update_most_wanted.plwiki_p.out -l sqlprocs-s2=1 /home/magnus/cronjobs/update_most_wanted.pl plwiki_p Running this script, or any of its individual lines, directly from shell, works absolutely fine. However, on I get (in the error file): /home/magnus/cronjobs/most_wanted_wrapper.sh[2]: 7274 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[3]: 7289 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[4]: 7290 Segmentation Fault(coredump) /home/magnus/cronjobs/most_wanted_wrapper.sh[5]: 7298 Segmentation Fault(coredump) I tried replacing qsub with cronsub, only to be met with a flood of errors relating to all the options it doesn't know, including the sql slot reservation (I guess that's my punishment for trying to be a good citizen...). Removing basically all options, using only lines like this: cronsub -s mw_dewiki_p /home/magnus/cronjobs/update_most_wanted.pl dewiki_p in the wrapper script, again runs peachy from shell, but barfs coredumps again when run via cronsub. So, as far as I am concerned, it's broken. Cheers, Magnus

On Mon, Feb 28, 2011 at 11:37 AM, River Tarnell <r.tarnell(a)ieee.org> wrote: qsub: Unknown option qsub: Unknown option qsub: Unknown option

On Mon, Feb 28, 2011 at 1:59 PM, River Tarnell &lt;r.tarnell(a)ieee.org&gt; wrote: + . /sge62/default/common/settings.sh + SGE_ROOT=/sge62 + export SGE_ROOT + + /sge62/util/arch ARCH=sol-amd64 + + /sge62/util/arch -m DEFAULTMANPATH=/usr/share/man + + /sge62/util/arch -mt MANTYPE=man + SGE_CELL=default + export SGE_CELL + SGE_CLUSTER_NAME=toolserver + export SGE_CLUSTER_NAME + unset SGE_QMASTER_PORT + unset SGE_EXECD_PORT + [ /sge62/man:/opt/ts/gnu/share/man:/opt/ts/share/man:/opt/SUNWspro/man:/usr/postgres/8.3/man:/opt/ts/mysql/5.1/share/man:/opt/ts/python/2.6/share/man:/opt/ts/perl/5.10/share/man:/opt/ts/ruby/1.8/share/man:/usr/share/man:/usr/sfw/share/man:/usr/cluster/man:/opt/jobserver/man = ] + MANPATH=/sge62/man:/sge62/man:/opt/ts/gnu/share/man:/opt/ts/share/man:/opt/SUNWspro/man:/usr/postgres/8.3/man:/opt/ts/mysql/5.1/share/man:/opt/ts/python/2.6/share/man:/opt/ts/perl/5.10/share/man:/opt/ts/ruby/1.8/share/man:/usr/share/man:/usr/sfw/share/man:/usr/cluster/man:/opt/jobserver/man + export MANPATH + PATH=/sge62/bin/sol-amd64:/sge62/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin + export PATH + unset ARCH DEFAULTMANPATH MANTYPE + PATH=/usr/bin:/sge62/bin/sol-amd64:/sge62/bin/sol-amd64:/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.12/bin:/opt/ts/perl/5/bin:/opt/ts/python/2.7/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.8/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/usr/cluster/bin:/opt/jobserver/bin + sflag= + QUEUE=all.q + OPTS= + hrt= + getopts slh f + shift 0 + [[ 2 -ge 1 ]] + JOBNAME=update_dupes + shift + [ -z ] + [[ 1 -ge 1 ]] + SCRIPT=/home/magnus/update_dupes.pl + + mktemp TMPF=/tmp/tmp.5jaiw2 + chmod 700 /tmp/tmp.5jaiw2 + trap rm -f /tmp/tmp.5jaiw2 0 + cat + 1> /tmp/tmp.5jaiw2 0<< #! /bin/sh cd $HOME exec "$@" exit 1 + grep ^#\$ + 0< /home/magnus/update_dupes.pl 1>> /tmp/tmp.5jaiw2 + grep Full jobname:[ \t]*update_dupes$ + 1> /dev/null + qstat -r + qsub -q all.q -N update_dupes /tmp/tmp.5jaiw2 /home/magnus/update_dupes.pl + 1> /dev/null qsub: Unknown option qsub: Unknown option qsub: Unknown option + rm -f /tmp/tmp.5jaiw2

On Mon, Feb 28, 2011 at 2:28 PM, Marcin Cieslak &lt;saper(a)saper.info&gt; wrote: Thanks! Wow, what a mess. Why does cronsub/qsub try to divine hidden vodoo meaning from my scripts (which predate qsub availability) anyway? If that is desired for some cases, why is that activated by default? Is there at least a qsub option to turn it off? Will that option work with cronsub? In my "day job", I work with a large compute farm (lfs, ~10K cores), so I'm more than happy to work with, and migrate existing cronjobs to, this system, but it would help if it weren't a deliberate minefield... Cheers, Magnus

There is another option that's quite useful: qsub -b y path-to-program This causes path-to-program (use full path like /usr/bin/something or path relative to your $HOME or it won't work) to be executed as a "binary" program. Your script will not be copied to a temporary directory on the SGE cluster and will be ran in place. Using "-b y" for Python scripts fixes the python import issue mentioned earlier on this list http://thread.gmane.org/gmane.org.wikimedia.toolserver/3069 (item two) Of course then job submission does not attempt to interpret your script in any way (including hidden voodoo to read qsub parameters from it), so "#$" won't work. Shell scripts work fine using "-b y". //Marcin

Have a look at /opt/local/bin/cronsub, it's not a very complicated script. Cronsub does not currently allow for adding custom qsub parameters. But what you can of course use own modified copy or whatever. cronsub currently copies the contents of the script to the temporary file that will be executed and copied by SGE, leaving original script intact in-place. -b y has its disadvantages, though. When I execute the program (I run a binary C program for test) and it abruptly stops by dumping core without generating any output. I combine "-b y" with "-m ae" to receive a mail when the job is completed, in case of crash it's reported in the e-mail: To: saper(a)clematis.toolserver.org Subject: Job 211410 (howto2) Aborted Date: Mon, 28 Feb 2011 20:57:51 +0000 (UTC) From: root(a)wolfsbane.toolserver.org (Super-User) Job 211410 (howto2) Aborted Exit Status = 139 Signal = SEGV User = saper Queue = all.q(a)wolfsbane.toolserver.org Host = wolfsbane.toolserver.org Start Time = 02/28/2011 20:57:50 End Time = 02/28/2011 20:57:51 CPU = 00:00:00 Max vmem = NA failed assumedly after job because: job 211410.1 died through signal SEGV (11) With shell scripts it is much more obvious, since shells complain to stderr (as in your case at least you could see the message). //Marcin

On Mon, Feb 28, 2011 at 3:24 PM, River Tarnell &lt;r.tarnell(a)ieee.org&gt; wrote: And more complicated than passing these parameters to qsub directly. Thankfully, Marcin was actually helpful in answering this. Your attempt to distract from the actual issues at hand by "literal interpretation" has been duly noted. I understand that, and I didn't say lfs was better (it has plenty of its own quirks), or that SGE was a bad choice - it seems to run well enough, all things considered. It's just that things like "SGE might interpret comments in your scripts as commands and die" or "you can't run qsub from within qsub" are types of behaviour that are unusual to say the least, and the toolserver users are left to the mercy of these issues (I'd call them bugs) without warning or documentation. (Yes, there is a wiki pages that says it can interpret special comments in your script. That's somewhat different from "it will reinterpret your Perl script and fall over"). Magnus

Hi, is there an offical list with all templates for disambiguations of all projects at the toolserver? I found only this page: https://fisheye.toolserver.org/browse/~raw,r=8465/pywikipedia/branches/rewr… Maybe there exist a page in the Mediawiki:API? But with this request I don't found this special templates. http://de.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop… Thanks for your help. Stefan (sk)

Hmm, but I will get the templates with a script. And when I show: http://en.wikipedia.org/wiki/MediaWiki:Disambiguationspage http://de.wikipedia.org/wiki/MediaWiki:Disambiguationspage http://fr.wikipedia.org/wiki/MediaWiki:Disambiguationspage then I can't see an standard format in this pages. Is there an other possibility? Stefan Am 28.02.2011 19:56, schrieb Steef389:

On Mon, Feb 28, 2011 at 7:57 PM, Stefan Kühn &lt;kuehn-s(a)gmx.net&gt; wrote: For each language: select pl_title from pagelinks,page where page_title="Disambiguationspage" and page_namespace=8 and pl_from=page_id and pl_namespace=10; seems to work... Cheers, Magnus

Magnus Manske wrote: Having learned some lessons today, you should write improved documentation on the wiki. :-) MZMcBride

On 02/28/2011 01:00 PM, Magnus Manske wrote: Any parameter? I smell a line break encoding issue. Try making sure there are no carriage returns in your crontab or in any of the scripts involved. -- Ilmari Karonen

It seems like qsub dumps core when executed from inside of the SGE job. /sge62/default/spool/wolfsbane/job_scripts/210756[3]: 1253 Segmentation Fault(coredump) I have filed https://jira.toolserver.org/browse/TS-963 about this. Really strange. //Marcin