How to assign environment variables in parallel in bash
After some ruminations, I came up with an ugly workaround:
#!/bin/bash
proc1=$(mktemp)
proc2=$(mktemp)
proc3=$(mktemp)
/path/to/longprocess1 > "$proc1" &
pid1=$!
/path/to/longprocess2 > "$proc2" &
pid2=$!
/path/to/longprocess3 > "$proc3" &
pid3=$!
wait "$pid1" "$pid2" "$pid3"
export var1="<("$proc1")"
export var2="<("$proc2")"
export var3="<("$proc3")"
rm -f "$proc1" "$proc2" "$proc3"
As requested in a comment, here is how to make this more extensible for an arbitrarily large list:
#!/bin/bash
declare -a pids
declare -a data
declare -a output
declare -a processes
# Generate the list of processes for demonstrative purposes
processes+=("/path/to/longprocess1")
processes+=("/path/to/longprocess2")
processes+=("/path/to/longprocess3")
index=0
for process in "${processes[@]}"; do
output+=("$(mktemp")
$process > ${output[$index]} &
pids+=("$!")
index=$((index+1))
done
wait ${pids[@]}
index=0
for process in "${processes[@]}"; do
data+="$(<"${output[index]}")"
rm -f "${output[index]}"
index=$((index+1))
done
export data
The resultant output will be in the data
array.
If you have more jobs than can safely be run in parallel at the same time, you can use parset from GNU Parallel:
parset foo,fizz,rick somecommand ::: bar baz morty
export foo
export fizz
export rick
See details: https://www.gnu.org/software/parallel/parset.html