How process substitution is implemented in bash?
Well, there are many aspects to it.
File descriptors
For each process, the kernel maintains a table of open files (well, it might be implemented differently, but since you are not able to see it anyways, you can just assume it's a simple table). That table contains information about which file it is/where it can be found, in which mode you opened it, at which position you are currently reading/writing, and whatever else is needed to actually perform I/O operations on that file. Now the process never gets to read (or even write) that table. When the process opens a file, it gets back a so-called file descriptor. Which is simply an index into the table.
The directory /dev/fd
and its content
On Linux dev/fd
is actually a symbolic link to /proc/self/fd
. /proc
is a pseudo file system in which the kernel maps several internal data structures to be accessed with the file API (so they just look like regular files/directories/symlinks to the programs). Especially there's information about all processes (which is what gave it the name). The symbolic link /proc/self
always refers to the directory associated with currently running process (that is, the process requesting it; different processes therefore will see different values). In the process's directory, there's a subdirectory fd
which for each open file contains a symbolic link whose name is just the decimal representation of file descriptor (the index into the process's file table, see previous section), and whose target is the file it corresponds to.
File descriptors when creating child processes
A child process is created by a fork
. A fork
makes a copy of the file descriptors, which means that the child process created has the very same list of open files as the parent process does. So unless one of the open files is closed by the child, accessing an inherited file descriptor in the child will access the very same file as accessing the original file descriptor in the parent process.
Note that after a fork, you initially have two copies of the same process which differ only in the return value from the fork call (the parent gets the PID of the child, the child gets 0). Normally, a fork is followed by an exec
to replace one of the copies by another executable. The open file descriptors survive that exec. Note also that before the exec, the process can do other manipulations (like closing files that the new process should not get, or opening other files).
Unnamed pipes
An unnamed pipe is just a pair of file descriptors created on request by the kernel, so that everything written to the first file descriptor is passed to the second. The most common use is for the piping construct foo | bar
of bash
, where the standard output of foo
is replaced by the write part of the pipe, and the standard input is replaces by the read part. Standard input and standard output are just the first two entries in the file table (entry 0 and 1; 2 is standard error), and therefore replacing it means just rewriting that table entry with the data corresponding to the other file descriptor (again, the actual implementation may differ). Since the process cannot access the table directly, there's a kernel function to do that.
Process substitution
Now we have everything together to understand how the process substitution works:
- The bash process creates an unnamed pipe for communication between the two processes created later.
- Bash forks for the
echo
process. The child process (which is an exact copy of the originalbash
process) closes the reading end of the pipe and replaces its own standard output with the writing end of the pipe. Given thatecho
is a shell builtin,bash
might spare itself theexec
call, but it doesn't matter anyway (the shell builtin might also be disabled, in which case it execs/bin/echo
). - Bash (the original, parent one) replaces the expression
<(echo 1)
by the pseudo file link in/dev/fd
referring to the reading end of the unnamed pipe. - Bash execs for the PHP process (note that after the fork, we are still inside [a copy of] bash). The new process closes the inherited write end of the unnamed pipe (and does some other preparatory steps), but leaves the read end open. Then it executed PHP.
- The PHP program receives the name in
/dev/fd/
. Since the the corresponding file descriptor is still open, it still corresponds to the reading end of the pipe. Therefore if the PHP program opens the given file for reading, what it actually does is to create asecond
file descriptor for the reading end of the unnamed pipe. But that's no problem, it could read from either. - Now the PHP program can read the reading end of the pipe through the new file descriptor, and thus receive the standard output of the
echo
command which goes to the writing end of the same pipe.
Borrowing from celtschk
's answer, /dev/fd
is a symbolic link to /proc/self/fd
. And /proc
is a pseudo filesystem, that presents information about processes and other system information in a hierarchical file-like structure. Files in /dev/fd
correspond to files, opened by a process and has file descriptor as their names and files themselves as their targets. Opening the file /dev/fd/N
is equivalent to duplicating descriptor N
(assuming that descriptor N
is open).
And here are results of my investigation of how it works (strace
output is rid of unnecessary details and modified to better express what's happening):
$ cat 1.c
#include <unistd.h>
#include <fcntl.h>
int main(int argc, char *argv[])
{
char buf[100];
int fd;
fd = open(argv[1], O_RDONLY);
read(fd, buf, 100);
write(STDOUT_FILENO, buf, n_read);
return 0;
}
$ gcc 1.c -o 1.out
$ cat 2.c
#include <unistd.h>
#include <string.h>
int main(void)
{
char *p = "hello, world\n";
write(STDOUT_FILENO, p, strlen(p));
return 0;
}
$ gcc 2.c -o 2.out
$ strace -f -e pipe,fcntl,dup2,close,clone,close,execve,wait4,read,open,write bash -c './1.out <(./2.out)'
[bash] pipe([3, 4]) = 0
[bash] dup2(3, 63) = 63
[bash] close(3) = 0
[bash] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7c211fb9d0) = p2
Process p2 attached
[bash] close(4) = 0
[bash] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7c211fb9d0) = p1
Process p1 attached
[bash] close(63) = 0
[p2] dup2(4, 1) = 1
[p2] close(4) = 0
[p2] close(63) = 0
[bash] wait4(-1, <unfinished ...>
Process bash suspended
[p1] execve("/home/yuri/_/1.out", ["/home/yuri/_/1.out", "/dev/fd/63"], [/* 31 vars */]) = 0
[p2] clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7c211fb9d0) = p22
Process p22 attached
[p22] execve("/home/yuri/_/2.out", ["/home/yuri/_/2.out"], [/* 31 vars */]) = 0
[p2] wait4(-1, <unfinished ...>
Process p2 suspended
[p1] open("/dev/fd/63", O_RDONLY) = 3
[p1] read(3, <unfinished ...>
[p22] write(1, "hello, world\n", 13) = 13
[p1] <... read resumed> "hello, world\n", 100) = 13
Process p2 resumed
Process p22 detached
[p1] write(1, "hello, world\n", 13) = 13
hello, world
[p2] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = p22
[p2] --- SIGCHLD (Child exited) @ 0 (0) ---
[p2] wait4(-1, 0x7fff190f289c, WNOHANG, NULL) = -1 ECHILD (No child processes)
Process bash resumed
Process p1 detached
[bash] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = p1
[bash] --- SIGCHLD (Child exited) @ 0 (0) ---
Process p2 detached
[bash] wait4(-1, 0x7fff190f2bdc, WNOHANG, NULL) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
[bash] wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = p2
[bash] wait4(-1, 0x7fff190f299c, WNOHANG, NULL) = -1 ECHILD (No child processes)
Basically, bash
creates a pipe and passes its ends to its children as file descriptors (read end to 1.out
, and write end to 2.out
). And passes read end as a command line parameter to 1.out
(/dev/fd/63
). This way 1.out
is able to open /dev/fd/63
.