Why is `tac file | grep foo' (piping) faster than `grep foo < <(tac file)' (process substitution)?
The construction <(tac file) causes to shell to:
- Create a pipe with a name
- On systems such as Linux and SysV which have
/dev/fd, a regular pipe is used, and/dev/fd/<the-file-descriptor-of-the-pipe>is used as the name. - On other systems, a named pipe is used, which requires creating an actual file entry on disk.
- On systems such as Linux and SysV which have
- Launch the command
tac fileand connect it to one end of the pipe. - Replace the whole construction on the command line with the name of the pipe.
After the replacement, the command line becomes:
grep whatever < /tmp/whatever-name-the-shell-used-for-the-named-pipe
And then grep is executed, and it reads its standard input (which is the pipe), reads it, and searches for its first argument in that.
So the end result is the same as with...
tac file | grep whatever
...in that the same two programs are launched and a pipe is still used to connect them. But the <( ... ) construction is more convoluted because it involves more steps and may involve a temporary file (the named pipe).
The <( ... ) construct is an extension, and is not available in the standard POSIX bourne shell nor on platforms that do not support /dev/fd or named pipes. For this reason alone, because the two alternatives being considered are exactly equivalent in functionality, the more portable command | other-command form is a better choice.
The <( ... ) construction should be slower because of the additional convolution, but it's only in the startup phase and I don't expect the difference to be easily measurable.
NOTE: On Linux SysV platforms, < ( ... ) does not use named pipes but instead uses regular pipes. Regular pipes (indeed all file descriptors) can be referred to by the special named /dev/fd/<file-descriptor-number so that's what the shell uses as a name for the pipe. In this way it avoids creating a real named pipe with a bona fide temporary filename in the real filesystem. Although the /dev/fd trick is what was used to implement this feature when it originally appears in ksh, it is an optimization: on platforms that don't support this, a regular named pipe in the real filesystem is used as described above.
ALSO NOTE: To describe the syntax as <<( ... ) is misleading. In fact it's <( ... ), which is replaced with the name of a pipe, and then the other < character which prefixes the whole thing is separate from this syntax and it's the regular well-known syntax for redirecting input from a file.
What is difference between | and <<()?
There is a difference between them:
|cause each command run in a separated subshell.<()run the command, which is substituted in background.
For the next two question, we will do some strace:
pipe:
$ strace -fc bash -c 'tac /usr/share/dict/american-english | grep qwerty'
$ time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.008120 2707 3 1 wait4
0.00 0.000000 0 352 read
0.00 0.000000 0 229 write
0.00 0.000000 0 20 2 open
0.00 0.000000 0 29 2 close
0.00 0.000000 0 40 17 stat
0.00 0.000000 0 19 fstat
0.00 0.000000 0 117 lseek
0.00 0.000000 0 38 mmap
0.00 0.000000 0 18 mprotect
0.00 0.000000 0 6 munmap
0.00 0.000000 0 25 brk
0.00 0.000000 0 22 rt_sigaction
0.00 0.000000 0 18 rt_sigprocmask
0.00 0.000000 0 1 rt_sigreturn
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 24 12 access
0.00 0.000000 0 1 pipe
0.00 0.000000 0 2 dup2
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 1 getpeername
0.00 0.000000 0 2 clone
0.00 0.000000 0 3 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 getrlimit
0.00 0.000000 0 13 getuid
0.00 0.000000 0 13 getgid
0.00 0.000000 0 13 geteuid
0.00 0.000000 0 13 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 3 arch_prctl
0.00 0.000000 0 1 time
------ ----------- ----------- --------- --------- ----------------
100.00 0.008120 1034 37 total
Process Substitution:
$ strace -fc bash -c 'grep qwerty < <(tac /usr/share/dict/american-english)'
$ time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.14 0.016001 4000 4 2 wait4
0.46 0.000075 0 229 write
0.24 0.000038 0 341 read
0.16 0.000026 1 24 brk
0.00 0.000000 0 21 2 open
0.00 0.000000 0 27 close
0.00 0.000000 0 40 17 stat
0.00 0.000000 0 19 fstat
0.00 0.000000 0 117 lseek
0.00 0.000000 0 38 mmap
0.00 0.000000 0 18 mprotect
0.00 0.000000 0 6 munmap
0.00 0.000000 0 35 rt_sigaction
0.00 0.000000 0 24 rt_sigprocmask
0.00 0.000000 0 2 rt_sigreturn
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 24 12 access
0.00 0.000000 0 1 pipe
0.00 0.000000 0 3 dup2
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 1 getpeername
0.00 0.000000 0 3 clone
0.00 0.000000 0 3 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 1 fcntl
0.00 0.000000 0 2 getrlimit
0.00 0.000000 0 13 getuid
0.00 0.000000 0 13 getgid
0.00 0.000000 0 13 geteuid
0.00 0.000000 0 13 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 3 arch_prctl
0.00 0.000000 0 1 time
------ ----------- ----------- --------- --------- ----------------
100.00 0.016140 1046 37 total
Why something faster than other?
And what is really faster?
You can see, process substitution is slower than pipe in this case, because it use more system call. Both spend much time to wait for child processes, but process substitution use more wait4() syscall, and use more time for each call than pipe.
Why no one suggest xargs ?
I don't think xargs can help anything here, this is not its jobs.
Update
As @Gilles's suggested, I do a test with bigger file, 2GB random data generated from /dev/urandom. It shows that pipe is really faster than process substitution.
pipe:
$ strace -fc bash -c 'tac sample.txt | grep qwerty'
$ time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
81.15 8.284959 2761653 3 1 wait4
17.89 1.825959 2 780959 read
0.91 0.092708 0 524286 write
0.05 0.005364 0 262146 lseek
0.00 0.000000 0 20 2 open
0.00 0.000000 0 29 2 close
0.00 0.000000 0 40 17 stat
0.00 0.000000 0 19 fstat
0.00 0.000000 0 38 mmap
0.00 0.000000 0 18 mprotect
0.00 0.000000 0 6 munmap
0.00 0.000000 0 25 brk
0.00 0.000000 0 22 rt_sigaction
0.00 0.000000 0 18 rt_sigprocmask
0.00 0.000000 0 1 rt_sigreturn
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 24 12 access
0.00 0.000000 0 1 pipe
0.00 0.000000 0 2 dup2
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 1 getpeername
0.00 0.000000 0 2 clone
0.00 0.000000 0 3 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 getrlimit
0.00 0.000000 0 13 getuid
0.00 0.000000 0 13 getgid
0.00 0.000000 0 13 geteuid
0.00 0.000000 0 13 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 3 arch_prctl
0.00 0.000000 0 1 time
------ ----------- ----------- --------- --------- ----------------
100.00 10.208990 1567727 37 total
process substitution:
$ strace -fc bash -c 'grep qwerty < <(tac sample.txt)'
$ time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.51 13.912869 3478217 4 2 wait4
0.38 0.053373 0 655269 read
0.09 0.013084 0 524286 write
0.02 0.002454 0 262146 lseek
0.00 0.000030 1 38 mmap
0.00 0.000024 1 24 12 access
0.00 0.000000 0 21 2 open
0.00 0.000000 0 27 close
0.00 0.000000 0 40 17 stat
0.00 0.000000 0 19 fstat
0.00 0.000000 0 18 mprotect
0.00 0.000000 0 6 munmap
0.00 0.000000 0 24 brk
0.00 0.000000 0 35 rt_sigaction
0.00 0.000000 0 24 rt_sigprocmask
0.00 0.000000 0 2 rt_sigreturn
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 1 pipe
0.00 0.000000 0 3 dup2
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 1 getpeername
0.00 0.000000 0 3 clone
0.00 0.000000 0 3 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 1 fcntl
0.00 0.000000 0 2 getrlimit
0.00 0.000000 0 13 getuid
0.00 0.000000 0 13 getgid
0.00 0.000000 0 13 geteuid
0.00 0.000000 0 13 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 3 arch_prctl
0.00 0.000000 0 1 time
------ ----------- ----------- --------- --------- ----------------
100.00 13.981834 1442060 37 total