Batch Processing with one task at a time

You can do this just using bash:

1- create fifo using mkfifo (you can create file too)

mkfifo mybuff

2- tail the fifo and pass each line to your executor (this will be kind of task server)

while IFS= read -r line
do
  echo "Running: '$line' with bash"
  bash -c "$line"
  echo "Finished '$line' with exit code: $?"
done < <(tail -f mybuff)

3- Send your commands to queue for example (this will client requesting tasks)

echo 'sleep 10' >> mybuff
echo 'echo "hello world"' >> mybuff
echo 'my fancy command to be executed' >> mybuff
...

PS: You can simplify step 2 like example below but this will finish task listener when you send "exit" to your command buffer

tail -f mybuff | bash -x

ts - task spooler. A simple unix batch system should be up for the task. It runs a task spooler/queue server and you can add a new task with a simple command like:

tsp yourcommand

You can specifify the number of slots - aka how many jobs should be executed at a time. By default this is set to one, AFAIK. It also has some advanced functionality, like running a command only if a certain previous command succeeds. You can view the queue and details for each job as well. You can find more details in the manpage.

Tags:

Batch Jobs