Is there a way to queue jobs on Toolforge https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_jobs? I want to run a number of one-off jobs (job1, job2, job3) in sequence. Job2 should start when job1 is done etc. They shouldn't run in parallell because they make requests to a service which can only take one request per second.
Making a YAML file with `wait: true` will make them run in sequence, but then I have a blocking process. Even if I could put it in the background I'm worried it'll stop at some point which means no more jobs would get started.
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
Hi Sebastian,
Is there a reason why you want to run different jobs instead of a single sh file containing a sequence of commands?
If you really, really need multiple jobs, I suggest looking at the --wait parameter [1] or, if you know the duration, just schedule them at convinient times so they don't overlap.
Best, Strainu
[1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_jobs
Pe vineri, 22 august 2025, Sebastian Berlin sebastian.berlin@wikimedia.se a scris:
Is there a way to queue jobs on Toolforge? I want to run a number of
one-off jobs (job1, job2, job3) in sequence. Job2 should start when job1 is done etc. They shouldn't run in parallell because they make requests to a service which can only take one request per second.
Making a YAML file with `wait: true` will make them run in sequence, but
then I have a blocking process. Even if I could put it in the background I'm worried it'll stop at some point which means no more jobs would get started.
Sebastian Berlin Utvecklare/Developer Wikimedia Sverige (WMSE)
E-post/E-Mail: sebastian.berlin@wikimedia.se Telefon/Phone: (+46) 0707 - 92 03 84
Yes, I run batches and want to keep the log files separate.
I tried putting all jobs in a YAML file with "wait" for each. As I was worried the process that started the jobs quit at some point, so only the first job ran.
*Sebastian Berlin* Utvecklare/*Developer* Wikimedia Sverige (WMSE)
E-post/*E-Mail*: sebastian.berlin@wikimedia.se Telefon/*Phone*: (+46) 0707 - 92 03 84
On Fri, 22 Aug 2025 at 16:46, Strainu strainu10@gmail.com wrote:
Hi Sebastian,
Is there a reason why you want to run different jobs instead of a single sh file containing a sequence of commands?
If you really, really need multiple jobs, I suggest looking at the --wait parameter [1] or, if you know the duration, just schedule them at convinient times so they don't overlap.
Best, Strainu
[1] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_jobs
Pe vineri, 22 august 2025, Sebastian Berlin sebastian.berlin@wikimedia.se a scris:
Is there a way to queue jobs on Toolforge? I want to run a number of
one-off jobs (job1, job2, job3) in sequence. Job2 should start when job1 is done etc. They shouldn't run in parallell because they make requests to a service which can only take one request per second.
Making a YAML file with `wait: true` will make them run in sequence, but
then I have a blocking process. Even if I could put it in the background I'm worried it'll stop at some point which means no more jobs would get started.
Sebastian Berlin Utvecklare/Developer Wikimedia Sverige (WMSE)
E-post/E-Mail: sebastian.berlin@wikimedia.se Telefon/Phone: (+46) 0707 - 92 03 84
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
wikitech-l@lists.wikimedia.org