[Labs-l] Changes to cron on tools

[Marc A. Pelletier]

Hey all,

We've had difficulty in the past with cron jobs not running for a number
of reasons (the primary being an overloaded bastion because of bots
running on it), so I made a few changes to how cron is to be used in the
future to vastly improve reliability:

a) cron for tools will now be executed from a different, dedicated
server that serves only that purpose.  This will insulate the bastions
from running cron jobs, and vice-versa.

b) the 'crontab' executable will edit the crontabs /on that host/
directly (so there will be no need to change how you're doing thing);
regardless of which bastion you run it from.  (So, from tools-login or
tools-dev, crontab will access the same schedule).

c) crontabs will be automatically edited to add a prefix to send your
jobs to the grid with reasonable default values unless the entry already
invokes jsub, jstart or qsub.  If the crontab program does that, it will
tell you so that you can inspect and edit the entries and adapt them to
your needs.

Those three measures will result in considerably more reliable execution
of scheduled jobs, as well as improved performance on the bastion hosts.

FOR THE NEXT WEEK OR SO:  both systems will be available at the same
time.  You can use the new system on the submit host by using 'xcrontab'
rather than 'crontab' to set or edit your crontab.  Be aware that /both/
schedules will be executed unless you also remove your bastion crontab.

AT THE END OF THAT WEEK:  any crontabs you may have had on tools-login
or tools-bastion will be moved to the new server, and 'crontab' will be
made to use the new system exclusively.  You have no intervention to
make; this will be done automatically.

-- Marc