Parallel execution of a program on multiple files
This is possible and does occur in reality. Use a lock file to avoid this situation. An example, from said page:
if mkdir /var/lock/mylock; then
echo "Locking succeeded" >&2
else
echo "Lock failed - exit" >&2
exit 1
fi
# ... program code ...
rmdir /var/lock/mylock
The two instances of your script can certainly interact in this way, causing the command to run twice. This is called a race condition.
One way to avoid this race condition would be if each instance grabbed its input file by moving it to another directory. Moving a file (inside the same filesystem) is atomic. Moving the input files may not be desirable, and this is already getting a bit complicated.
mkdir staging-$$ making-$$
for input in folder/*; do
name=${x#folder/}
staging=staging-$$/$name
output=making-$$/$name
destination=done/$name
if mv -- "$input" "$staging" 2>/dev/null; then
bin/myProgram "$staging" >"$output"
mv -- "$output" "$destination"
mv -- "$staging" "$input"
fi
done
A simple way to process the files in parallel using a widely-available tool is GNU make, using the -j
flag for parallel execution. Here's a makefile for this task (remember to use tabs to indent commands):
all: $(patsubst folder/%,done/%,$(wildcard folder/*))
done/%: folder/%
./bin/myProgram $< >[email protected]
mv [email protected] $@
Run make -j 3
to run 3 instances in parallel.
See also Four tasks in parallel... how do I do that?