Hello toolserver users,
as you may know, there were some bigger problems related to sun grid engine starting in november 2011. I asked DaB. to become a sge manager for helping them to solve these problems. During the last months i silently started reconfiguring sge in small steps so that it was always possible to use it as before and no downtime was needed. This took some time because i am only a volunteer and i had to changes nearly everything. Additional Nosy and DaB. changed some solaris configurations that i proposed.
All scripts that used grid engine before can continue to run without changes. But maybe you can increase your script performance by adding additional informations.
In the past you were requested to choose a suitable queue (all.q or longrun) for your job. Many people choosed a queue that did not fit best for their task. So i changed this procedure.
Now you have to add all resources that your job needs during runtime on job submition. Then sge will choose queue and host that fits best for your requirements. So you don't have to care about different queues anymore (you may have seen that there are much more queues than before).
All jobs must at least contain informations about maximum runtime (h_rt) and peak memory usage (virtual_free). This information may get obligatory in future. Currently only a warning message is shown. You also have to request other resources like sql connections, free temp space, etc. if these are needed by your job. Please read documentation on toolserverwiki i have updated today: https://wiki.toolserver.org/view/Job_scheduling This currently contains the main informations you need to know, but maybe i add some more examples later.
I also have added a new script called "qcronsub". This is the replacement for "cronsub" most of you used before. Differently to cronsub it accepts the same arguments as the original "qsub" command by grid engine. So now it is possible the add all resource values at command line.
Please note that you should always use cronie at submit.toolserver.org for submitting jobs to sge by cron. These cron tasks will always be executed even if one host (e.g. clematis or willow) is down. This is the suggested usage since about 17 months. Many people have migrated their cron jobs from nightshade to willow during the last weeks. But they will have the same problem again if willow must be shut down for a longer time (which hopefully never happens).
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03.03.2012 22:46, Merlissimo wrote:
Hello toolserver users,
as you may know, there were some bigger problems related to sun grid engine starting in november 2011. I asked DaB. to become a sge manager for helping them to solve these problems. During the last months i silently started reconfiguring sge in small steps so that it was always possible to use it as before and no downtime was needed. This took some time because i am only a volunteer and i had to changes nearly everything. Additional Nosy and DaB. changed some solaris configurations that i proposed.
All scripts that used grid engine before can continue to run without changes. But maybe you can increase your script performance by adding additional informations.
In the past you were requested to choose a suitable queue (all.q or longrun) for your job. Many people choosed a queue that did not fit best for their task. So i changed this procedure.
Now you have to add all resources that your job needs during runtime on job submition. Then sge will choose queue and host that fits best for your requirements. So you don't have to care about different queues anymore (you may have seen that there are much more queues than before).
All jobs must at least contain informations about maximum runtime (h_rt) and peak memory usage (virtual_free). This information may get obligatory in future. Currently only a warning message is shown. You also have to request other resources like sql connections, free temp space, etc. if these are needed by your job. Please read documentation on toolserverwiki i have updated today: https://wiki.toolserver.org/view/Job_scheduling This currently contains the main informations you need to know, but maybe i add some more examples later.
I also have added a new script called "qcronsub". This is the replacement for "cronsub" most of you used before. Differently to cronsub it accepts the same arguments as the original "qsub" command by grid engine. So now it is possible the add all resource values at command line.
Please note that you should always use cronie at submit.toolserver.org for submitting jobs to sge by cron. These cron tasks will always be executed even if one host (e.g. clematis or willow) is down. This is the suggested usage since about 17 months. Many people have migrated their cron jobs from nightshade to willow during the last weeks. But they will have the same problem again if willow must be shut down for a longer time (which hopefully never happens).
First thanks a lot for your effort!! First I thought why making it even more complicate (this increases the possibility of mal-config) but after setting my cron(ie)tab up I have to say it makes sense! Will be a good thing even thought not simple.
-- Example:
This morning Dr. Trigon complained that his job "mainbot" did not run immediatly and was queued for a long time. I would guess he submitted his job from cron using "cronsub mainbot -l /home/drtrigon/pywikipedia/mainbot.py" This indicates that the job runs forevery (longrun) with unkown memory usage. So grid engine was only able to start this job on willow. It is not possible to run infinite job on the webservers (only shorter jobs are allowed so that most jobs have finished before high webserver usage is expected during the evening). Nor it was possible to run it on the server running mail transfer agent which only have less than 500MB memory free, but much cpu power (expected memory usage is unkown). Other servers like nightshade and yarrow aren't currently available.
Thanks for taking me as an example - that help a lot... ;))
The exact command was: cronsub -sl mainbot $HOME/pywikipedia/bot_control.py -default -cron (very close... ;)
According to the last run of this job it takes about 2 hours and 30 minutes runtime and had a peek usage of 370 MB memory. I got these values by requesting grid engine about usage statistics of the last ten days: "qacct -j mainbot -d 10". To be safe that the job gets always enough resouces i would suggest to raise the values to 4 hours and 500MB memory. It is not a problem if you request more resouces than really needed, but job needing more resources than requested may be killed. So the new submit command would be:
"qcronsub -N mainbot -l h_rt=4:00:00 -l virtual_free=500MB /home/drtrigon/pywikipedia/mainbot.py"
I use now: qcronsub -l h_rt=12:00:00 -l virtual_free=500M -m a -b y -N mainbot $HOME/pywikipedia/bot_control.py -default -cron (a little bit more of time and '-m a -b y')
And here my key question arises; you mentioned 'qacct' to get more info (thanks for this hint) and this is one of the biggest problem I had with the whole SGE stuff; I was not able to get a complete docu whether on the toolserver nor else. At the moment, on the toolserver commands like 'qstat' or 'qdel' are not covered anymore. I (we) would like to know more about this great system.
E.g. what is the analogue to the old commands:
'cronsub [jobname] [command]' has become 'qcronsub -l h_rt=06:00:00 -l virtual_free=100M -m a -b y -N [jobname] [command]'
'cronsub -l [jobname] [command]' has become 'qcronsub -l h_rt=INFINITY -l virtual_free=100M -m a -b y -N [jobname] [command]'
as far as I can see... (do not remember what the '-s' was for...) Is this correct?
Thanks for your work Merlissimo and greetings DrTrigon
Dr. Trigon wrote:
On 03.03.2012 22:46, Merlissimo wrote:
Hello toolserver users,
[...]
First thanks a lot for your effort!! First I thought why making it even more complicate (this increases the possibility of mal-config) but after setting my cron(ie)tab up I have to say it makes sense! Will be a good thing even thought not simple.
-- Example:
This morning Dr. Trigon complained that his job "mainbot" did not run immediatly and was queued for a long time. I would guess he submitted his job from cron using "cronsub mainbot -l /home/drtrigon/pywikipedia/mainbot.py" This indicates that the job runs forevery (longrun) with unkown memory usage. So grid engine was only able to start this job on willow. It is not possible to run infinite job on the webservers (only shorter jobs are allowed so that most jobs have finished before high webserver usage is expected during the evening). Nor it was possible to run it on the server running mail transfer agent which only have less than 500MB memory free, but much cpu power (expected memory usage is unkown). Other servers like nightshade and yarrow aren't currently available.
Thanks for taking me as an example - that help a lot... ;))
The exact command was: cronsub -sl mainbot $HOME/pywikipedia/bot_control.py -default -cron (very close... ;)
According to the last run of this job it takes about 2 hours and 30 minutes runtime and had a peek usage of 370 MB memory. I got these values by requesting grid engine about usage statistics of the last ten days: "qacct -j mainbot -d 10". To be safe that the job gets always enough resouces i would suggest to raise the values to 4 hours and 500MB memory. It is not a problem if you request more resouces than really needed, but job needing more resources than requested may be killed. So the new submit command would be:
"qcronsub -N mainbot -l h_rt=4:00:00 -l virtual_free=500MB /home/drtrigon/pywikipedia/mainbot.py"
I use now: qcronsub -l h_rt=12:00:00 -l virtual_free=500M -m a -b y -N mainbot $HOME/pywikipedia/bot_control.py -default -cron (a little bit more of time and '-m a -b y')
And here my key question arises; you mentioned 'qacct' to get more info (thanks for this hint) and this is one of the biggest problem I had with the whole SGE stuff; I was not able to get a complete docu whether on the toolserver nor else. At the moment, on the toolserver commands like 'qstat' or 'qdel' are not covered anymore. I (we) would like to know more about this great system.
E.g. what is the analogue to the old commands:
'cronsub [jobname] [command]' has become 'qcronsub -l h_rt=06:00:00 -l virtual_free=100M -m a -b y -N [jobname] [command]'
'cronsub -l [jobname] [command]' has become 'qcronsub -l h_rt=INFINITY -l virtual_free=100M -m a -b y -N [jobname] [command]'
In both cases the old behavior was without -m a -b y, so
'cronsub [jobname] [command]' has become 'qcronsub -l h_rt=06:00:00 -l virtual_free=100M -N [jobname] [command]
'cronsub -l [jobname] [command]' has become 'qcronsub -l h_rt=INFINITY -l virtual_free=100M -N [jobname] [command]
The -b y option is mostly useful for binaries, e.g. if you don't submit the python script itself, but call the binary interpreter (python) with an argument. It is just an option if the submitted script file should be copied to a local filesystem on execution server (which increases performance, makes nfs error impossible and was always the default setting) or executed directly from your home (if you use -b y). In most cases this option isn't needed and copying is the best for most shell scripts.
I added this to the interwiki bots exmaple because 1: if you submit a job interwiki.py the file is copied the sge spool directory and the job is queued 2: then you update your local svn copy 3: afterwards the job is started Now the copied interwiki.py can be older than the rest of your pwd files. So in this case its better to use the same version of all pwd files directly from your homedir. It's very unlikey that this problem really happends, but i wanted to write a perfect example.
as far as I can see... (do not remember what the '-s' was for...) Is this correct?
The -s at cronsub is for merging standard error stream into standard output stream. This is now -j y. I don't know why river used -s for this (perhaps for stream?). Some weeks ago i installed a script that removes empty log files for standard error/output stream after job execution. Many people used this option to prevent that their homedir contain so many empty error logs.
Thanks for your work Merlissimo and greetings DrTrigon
Merlissimo
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04.03.2012 01:06, Merlissimo wrote:
In both cases the old behavior was without -m a -b y, so
'cronsub [jobname] [command]' has become 'qcronsub -l h_rt=06:00:00 -l virtual_free=100M -N [jobname] [command]
'cronsub -l [jobname] [command]' has become 'qcronsub -l h_rt=INFINITY -l virtual_free=100M -N [jobname] [command]
The -b y option is mostly useful for binaries, e.g. if you don't submit the python script itself, but call the binary interpreter (python) with an argument. It is just an option if the submitted script file should be copied to a local filesystem on execution server (which increases performance, makes nfs error impossible and was always the default setting) or executed directly from your home (if you use -b y). In most cases this option isn't needed and copying is the best for most shell scripts.
Thanks for that info! I thought to remember once there was a comment about cronsub copying (like -b y does) the script...?!? Anyway I had to use '-b y' else my script closed directly after execution. What am I doing wrong here? (my script do read and write files in my home, e.g. config and log files - may be this is related?)
Greetings DrTrigon
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04.03.2012 01:06, Merlissimo wrote:
Some weeks ago i installed a script that removes empty log files for standard error/output stream after job execution. Many people used this option to prevent that their homedir contain so many empty error logs.
Could you may be explain how to use this script you installed?
Another question I have; Is there a way to finish and start a script without having to wait until a queue becomes free by re-using to old? E.g. if I like to restart my ircbot (running for INFTY time) I have to wait some time until the needed queue becomes available, but what about just re-using the queue it already had before?
Or is there an alternative way to have such a ircbot script running continuously without having to wait (e.g. more than 1 minute) during a restart? Can I trigger a restart (reading the new script from hdd) while keeping it in the queue?
Thanks again! Greetings DrTrigon
Dr. Trigon wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04.03.2012 01:06, Merlissimo wrote:
Some weeks ago i installed a script that removes empty log files for standard error/output stream after job execution. Many people used this option to prevent that their homedir contain so many empty error logs.
Could you may be explain how to use this script you installed?
That was https://jira.toolserver.org/browse/TS-1224. You don't have to care about it.
Another question I have; Is there a way to finish and start a script without having to wait until a queue becomes free by re-using to old? E.g. if I like to restart my ircbot (running for INFTY time) I have to wait some time until the needed queue becomes available, but what about just re-using the queue it already had before?
Or is there an alternative way to have such a ircbot script running continuously without having to wait (e.g. more than 1 minute) during a restart? Can I trigger a restart (reading the new script from hdd) while keeping it in the queue?
Thanks again! Greetings DrTrigon
After the job has finished its slot is used by another job. You can only write some trigger at your programm that automatically restarts itself. E.g. a shell program you submit to sge starts your bot. If the bot script exists its checks the return code and if this is a special value it starts your bot again within a loop.
But this should not be necessary because most of the time enough resources are available especially for jobs either needing low memory or have a short/medium runtime. Only jobs needing both (long runtime and much memory like interwikibots) may need to wait some time because they can onyl run on willow. But this will change when nigtshadeand and yarrow are back. Most jobs are submitted at the same time on the hour. So there is a peak which takes only very few minutes. The rest of the hour not so many jobs are submitted by cron.
I wonder how often an irc bot must be restartet. If it must be restarted regulary you should not define a infinite runtime.
Merlissimo
On 04/03/12 01:06, Merlissimo wrote:
In both cases the old behavior was without -m a -b y, so
(...)
The -b y option is mostly useful for binaries, e.g. if you don't submit the python script itself, but call the binary interpreter (python) with an argument. (...)
Actually, cronsub works as if providing -b y. It doesn't provide -b y, but submits a new script which then calls the calling script, so it's like submitting the script with -b y.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04.03.2012 16:25, Platonides wrote:
On 04/03/12 01:06, Merlissimo wrote:
In both cases the old behavior was without -m a -b y, so
(...)
The -b y option is mostly useful for binaries, e.g. if you don't submit the python script itself, but call the binary interpreter (python) with an argument. (...)
Actually, cronsub works as if providing -b y. It doesn't provide -b y, but submits a new script which then calls the calling script, so it's like submitting the script with -b y.
Aaa-ha! That explains a lot! Thanks for the hint! Greetings
On 04/03/12 00:17, Dr. Trigon wrote:
And here my key question arises; you mentioned 'qacct' to get more info (thanks for this hint) and this is one of the biggest problem I had with the whole SGE stuff; I was not able to get a complete docu whether on the toolserver nor else. At the moment, on the toolserver commands like 'qstat' or 'qdel' are not covered anymore. I (we) would like to know more about this great system.
? They are documented in the server man pages. Just run man qacct Or PAGER=less man qacct as I find it a nicer one.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 04.03.2012 01:21, Platonides wrote:
On 04/03/12 00:17, Dr. Trigon wrote:
And here my key question arises; you mentioned 'qacct' to get more info (thanks for this hint) and this is one of the biggest problem I had with the whole SGE stuff; I was not able to get a complete docu whether on the toolserver nor else. At the moment, on the toolserver commands like 'qstat' or 'qdel' are not covered anymore. I (we) would like to know more about this great system.
? They are documented in the server man pages. Just run man qacct Or PAGER=less man qacct as I find it a nicer one.
No this was not my point. If I know there is a command like e.g. 'qacct' I know how to get help. My question was where do I get a list of all SGE commands, options and backgrund infos? (it does not need to be a reference book, but a user manual would be nice)
I searched the net several times for SGE infos and found some in several places but that have been small parts of the whole docu only. But I assume there has to be something... ;)
Greetings DrTrigon
On 03/03/12 22:46, Merlissimo wrote:
All jobs must at least contain informations about maximum runtime (h_rt) and peak memory usage (virtual_free). This information may get obligatory in future. Currently only a warning message is shown. You also have to request other resources like sql connections, free temp space, etc. if these are needed by your job. Please read documentation on toolserverwiki i have updated today: https://wiki.toolserver.org/view/Job_scheduling This currently contains the main informations you need to know, but maybe i add some more examples later.
Thanks a lot Merlissimo. The new options look very good.
I also have added a new script called "qcronsub". This is the replacement for "cronsub" most of you used before. Differently to cronsub it accepts the same arguments as the original "qsub" command by grid engine. So now it is possible the add all resource values at command line.
I don't think qcronsub is an appropiate name for a qsub script which doesn't enqueue tasks twice. The name recalls the old cronsub but the interface is completely different. I'd call it qsub-unique or similar.
As the options to configure are now much more, and sge unintuitive argument names, I think it would be appropiate to have a wizard which provides generates for you the appropiate command line. (Bonus if it's a curses program which edits submit crontab directly)
All jobs must at least contain informations about maximum runtime (h_rt) and peak memory usage (virtual_free). This information may get obligatory in future. Currently only a warning message is shown. You also have to request other resources like sql connections, free temp space, etc. if these are needed by your job. Please read documentation on toolserverwiki i have updated today: https://wiki.toolserver.org/view/Job_scheduling This currently contains the main informations you need to know, but maybe i add some more examples later.
Thank you for your efforts. It would be sad to see SGE go because lack of proper maintenance. I already got used to it and currently I submit all my (even on-off) SQL queries by qsub.
I have one question that is vaguely related problem related to the virtual memory size.
Does anyone have some reasonable values for virtual memory limits (and other resources as well) for interwiki.py?
On tools.wikimedia.pl (a small toolserver box run by Wikimedia Poland) we run some interwiki bots and I am slowly getting crazy to figure out correct values for this. As far as I can see, for interwiki.py the sky is the limit.
Any hints how to gauge virtual memory usage given some set of input data? We run bots not only for wikipedia but also for wiktionary.
//Saper
By the way, did you have a chance to check how mail is delivered to job submitters?
We have tickets:
https://jira.toolserver.org/browse/TS-862 I don't receive SGE "job started/ended" mails anymore
and related (if not duplicated):
https://jira.toolserver.org/browse/TS-955 willow local MTA does not seem to accept mail from SGE jobs
There were some issues in configuring MTAs/SGE so that email works "just right", i.e. is delivered to the submitter (saper@submittedhost) and not to the localuser on the host running the task (saper@runninghost).
Another interesting issue (which probably can be solved only by software upgrade) is that attempt to submit a job from another job results in segmentation fault:
https://jira.toolserver.org/browse/TS-963 qsub segfaults when running from within SGE job
//Saper
toolserver-l@lists.wikimedia.org