Bash threading: wait for all job threads to finish doesn't work?

Update: good points in the comment.

So, on a second look, it turns out the problem is the subshell that is created by the pipe to the loop. It's a good way to structure the script but you need to do the final wait in the shell that spun off the background tasks.

So do something like this:

find /home/some/path -type f | (while read filename; do
    echo "Creating archive of $filename"
    func $somevariable &
  done
  wait
)

Tricky! The problem is that this block

find /home/some/path -type f | while read filename ; do
  ...
done

Creates a subshell. The func $somevariable jobs are created in that subshell. The parent shell sees that all the background jobs it created have finished, it doesn't keep track of background jobs created by subshells it spawned.

The easiest fix is to create your background jobs from the parent shell instead. You can avoid creating a subshell by not using a pipe:

while read filename ; do
  ...
done < <(find /home/some/path -type f)

Well, that creates a subshell---for the find---but the while block is no longer in a subshell.

Note that the above works only under bash. (Don't know about ksh or zsh, perhaps it works there too. But it won't work under ash and other sh derivatives.)

If you execute wait with no arguments, it is supposed to wait for currently active child processes to complete.

The problem is likely to be that "all currently active child processes" does not mean what you think it means in this context. In particular, if you create pipelines in a subshell it is not entirely clear if they would be waited for in the parent shell.

I suspect that wait actually only waits for processes / pipelines that show up in the output of jobs. Try some experiments ...

A possible alternative may be to capture the child process ids and do a wait n call for each id.