Running up to X commands in parallel
If you install the GNU Parallel tool you can make pretty easy work of what you're trying to accomplish:
$ find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | \
parallel --gnu -j 8 --workdir $PWD ' \
echo "Encrypting {}..."; \
gpg --trust-model always \
--recipient "[email protected]" --output "{}.gpg" \
--encrypt "{}" && rm "{}" \
'
details
The above is taking the output of find
and running it through to parallel
, and running 8 at a time. Everywhere there's an occurrence of {}
the filenames that are being passed through from find
will replace the {}
in those spots.
References
- Running shell script in parallel
You might want to look at gnu parallel and its --semaphore
option. From the documentation:
--semaphore
Work as a counting semaphore. --semaphore will cause GNU parallel to start command in the background. When the number of simultaneous jobs is reached, GNU parallel will wait for one of these to complete before starting another command.
You use --jobs 8
to limit the number of jobs to 8. You can pipe the output of sort into parallel
like you would do with xargs
.
sem
is an alias for parallel --semaphore