Bash script processing limited number of commands in parallel
See parallel. Its syntax is similar to xargs
, but it runs the commands in parallel.
Use the wait
built-in:
process1 &
process2 &
process3 &
process4 &
wait
process5 &
process6 &
process7 &
process8 &
wait
For the above example, 4 processes process1
... process4
would be started in the background, and the shell would wait until those are completed before starting the next set.
From the GNU manual:
wait [jobspec or pid ...]
Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero. If neither jobspec nor pid specifies an active child process of the shell, the return status is 127.