How to emulate Process Substitution in Dash?
You can reproduce what the shell does under the hood by doing the plumbing manually. If your system has /dev/fd/NNN
entries, you can use file descriptor shuffling: you can translate
main_command <(produce_arg1) <(produce_arg2) >(consume_arg3) >(consume_arg4)
to
{ produce_arg1 |
{ produce_arg2 |
{ main_command /dev/fd5 /dev/fd6 /dev/fd3 /dev/fd4 </dev/fd/8 >/dev/fd/9; } 5<&0 3>&1 |
consume_arg3; } 6<&0 4>&1; |
consume_arg4; } 8<&0 9>&1
I've shown a more complex example to illustrate multiple inputs and outputs. If you don't need to read from standard input, and the only reason you're using process substitution is that the command requires an explicit file name, you can simply use /dev/stdin
:
main_command <(produce_arg1)
produce_arg1 | main_command /dev/stdin
Without /dev/fd/NNN
, you need to use a named pipe. A named pipe is a directory entry, so you need to create a temporary file somewhere, but that file is just a name, it doesn't contain any data.
tmp=$(mktemp -d)
mkfifo "$tmp/f1" "$tmp/f2" "$tmp/f3" "$tmp/f4"
produce_arg1 >"$tmp/f1" &
produce_arg2 >"$tmp/f2" &
consume_arg3 <"$tmp/f3" &
consume_arg4 <"$tmp/f4" &
main_command "$tmp/f1" "$tmp/f2" "$tmp/f3" "$tmp/f4"
rm -r "$tmp"
The question in the current bounty notice:
the general example is too complicated. Can somebody please explain how to implement following example?
diff <(cat "$2" | xz -d) <(cat "$1" | xz -d)
seems to have an answer here.
As shown in Gilles' answer, the general idea is to send the output of "producer" commands to new device files1 at different stages of a pipeline, making them available to "consumer" commands, which may possibly take file names as arguments (assuming that your system gives you access to file descriptors as /dev/fd/X
).
The simplest way to achieve what you are looking for is probably:
xz -cd file1.xz | { xz -cd file2.xz | diff /dev/fd/3 -; } 3<&0
(Using file1.xz
in place of "$1"
for readability and xz -cd
instead of cat ... | xz -d
because a single command is enough).
The output of the first "producer" command, xz -cd file1.xz
, is piped to a compound command ({...}
); but, instead of being consumed immediately as the standard input of the next command, it is duplicated to file descriptor 3
and thus made accessible to everything inside the compound command as /dev/fd/3
. The output of the second "producer" command, xz -cd file2.xz
, which does consume neither its standard input nor anything from file descriptor 3
, is then piped to the "consumer" command diff
, which reads from its standard input and from /dev/fd/3
.
Piping and file descriptor duplication may be added in order to provide device files for as many "producer" commands as needed, e.g.:
xz -cd file1.xz | { xz -cd file2.xz | { diff /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0
While it may be irrelevant in the context of your specific question, it is worth noting that:
cmd1 <(cmd2) <(cmd3)
,cmd2 | { cmd3 | { cmd1 /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0
and( cmd2 | ( cmd3 | ( cmd1 /dev/fd/3 /dev/fd/4 ) 4<&0 ) 3<&0 )
have different potential effects on the initial execution environment.Contrary to what happens in
cmd1 <(cmd2) <(cmd3)
,cmd3
andcmd1
incmd2 | { cmd3 | { cmd1 /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0
will not be able to read any input from the user. That will require further file descriptors. For instance, to matchdiff <(echo foo) <(read var; echo "$var")
you will need something like
{ echo foo | { read var 0<&9; echo "$var" | diff /dev/fd/3 -; } 3<&0; } 9<&0
1 More on them can be found on U&L, e.g. in Understanding /dev and its subdirs and files.
Can somebody please explain how to implement following example?
diff <(cat "$2" | xz -d) <(cat "$1" | xz -d)
I think named pipes are simpler to grok than redirections, so in simplest terms:
mkfifo p1 p2 # create pipes
cat "$2" | xz -d > p1 & # start the commands in the background
cat "$1" | xz -d > p2 & # with output to the pipes
diff p1 p2 # run 'diff', reading from the pipes
rm p1 p2 # remove them at the end
p1
and p2
are temporary named pipes, they could be named anything.
It would be smarter to create the pipes in /tmp
, e.g. in a directory as Gilles' answer shows, and note that you don't need cat
here, so:
tmpdir=$(mktemp -d)
mkfifo "$tmpdir/p1" "$tmpdir/p2"
xz -d < "$2" > "$tmpdir/p1" &
xz -d < "$1" > "$tmpdir/p2" &
diff "$tmpdir/p1" "$tmpdir/p2"
rm -r "$tmpdir"
(You could probably get away without the quotes, too, since mktemp
is likely to create "nice" filenames.)
The pipe solution does have the caveat that if the main command (diff
) dies before reading all the input, the background processes can be left hanging.