Jayndoodle Jayndoodle - 5 months ago 22
Linux Question

Running many small scripts throughout the day with cron

I have a script, lets call it run-jobs.sh that I want to run. It takes in 1 argument. What I currently have set up is a cron

1 7 * * * /home/admin/run-jobs.sh arg1
2 7 * * * /home/admin/run-jobs.sh arg2
3 7 * * * /home/admin/run-jobs.sh arg3


I have many different arguments (lets say 100) to run, but I want to spread it throughout the day so it doesn't just run everything all at once. What is the best way to do this with cron, without having to make a huge cron list and have to edit it everytime? Is there something other than cron that will help me with this problem?

Answer Source

The easiest thing might be to just have a wrapper script that runs once a day and sleeps between invocations, e.g.

crontab:

0 0 * * * /home/admin/run-all-jobs.sh arg1 arg2 arg3

run-all-jobs.sh:

for arg in "$@"; do
  # run the job as a background task,
  # so the job's duration doesn't throw off the timing
  run-jobs.sh "$arg" &
  # split the args evenly around 24 hours
  sleep $(( 24 * 60 * 60 / $# ))
done

This is easy and for simple cases will work just fine, but it's not very robust. Cron handles a lot of edge-cases for you.


Another option would be to dynamically generate the crontab, so you don't have to manually edit it. This example schedules each job at 15-minute intervals so it will start overlapping jobs once you have more than 96, but you can tweak it to schedule the jobs at different intervals.

generate-jobs.sh:

for (( i=1; i<=$#; i++ )); do
  printf "%d %d * * * /home/admin/run-jobs.sh '%s'\n" \
      "$(( i / 4 ))" "$(( i % 4 ))" "${!i}"
done > /etc/cron.d/run-jobs.cron

This writes a file to /etc/cron.d, which cron will read from like your regular crontab.


A third option, which is a little more involved but possibly what you really want, is to use at. at is a great tool that lets you schedule tasks "at" some point in the future. With at you can avoid cron altogether, and just schedule your jobs whenever you'd like.