How does a FIFO (named pipe) differs from a regular pipe (unnamed pipe)?

"Named pipe" is actually a very accurate name for what it is — it is just like a regular pipe, except that it has a name (on a filesystem).

A pipe — the regular, un-named ("anonymous") one used in some-command | grep pattern is a special kind of file. And I do mean file, you read and write to it just like you do every other file. Grep doesn't really care¹ that it's reading from a pipe instead of a terminal³ or an ordinary file.

Technically, what goes on behind the scenes is that stdin, stdout, and stderr are three open files (file descriptors) passed to every command run. File descriptors (which are used in every syscall to read/write/etc. files) are just numbers; stdin, stdout, and stderr are file descriptors 0, 1, and 2. So when your shell sets up some-command | grep what it does is something this:

  1. Asks the kernel for an anonymous pipe. There is no name, so this can't be done with open like for a normal file — instead it's done with pipe or pipe2, which returns two file descriptors.⁴

  2. Forks off a child process (fork() creates a copy of the parent process; both sides of the pipe are open here), copies the write-side of the pipe to fd 1 (stdout). The kernel has a syscall to copy around file descriptor numbers; it's dup2() or dup3(). It then closes the read side and other copy of the write side. Finally, it uses execve to execute some-command. Since the pipe is fd 1, stdout of some-command is the pipe.

  3. Forks of another child process. This time, it duplicates the read side of the pipe to fd 0 (stdin), and executes grep. So grep will read from the pipe as stdin.

  4. Then it waits for both of those children to exit.

  5. At this point, the kernel notices that the pipe isn't open any more, and garbage collects it. That's what actually destroys the pipe.

A named pipe just gives that anonymous pipe a name by putting it in the filesystem. So now any process, at any point in the future, can obtain a file descriptor for the pipe by using an ordinary open syscall. Conceptually, the pipe won't be destroyed until both all readers/writers have closed it and it's unlinked from the filesystem.²

This, by the way, is how files in general work on Unix. unlink (the syscall behind rm) just removes one of the names for the file; only when all names are removed and nothing has the file open will it actually be deleted. A couple of answers here explore this:

  • Why do hard links seem to take the same space as the originals?
  • How can a log program continue to log to a deleted file?
  • What is Linux doing differently that allows me to remove/replace files where Windows would complain the file is currently in use?

FOOTNOTES

  1. Technically this probably isn't true — it's probably possible to do some optimizations by knowing, and actual grep implementations have often been heavily optimized. But conceptually it doesn't care (and indeed a straightforward implementation of grep wouldn't).
  2. Of course the kernel doesn't actually keep all the data structures around in memory forever, but rather it recreates them, transparently, whenever the first program opens the named pipe (and then keeps them as long as its open). So it's as if they existed as long as the name does.
  3. Terminal isn't a common place for grep to read from, but it's the default stdin when you don't specify another. So if you type just grep pattern in to your shell, grep will be reading from the terminal. The only use for this that comes to mind is if you're about to paste something to the terminal.
  4. On Linux, anonymous pipes actually are created on a special filesystem, pipefs. See How pipes work in Linux for details. Note that this is an internal implementation detail of Linux.

I think you're getting mixed up between shell syntax for pipelines vs. the underlying Unix systems programming. A pipe / FIFO is a type of file that isn't stored on disk, but instead passed data from a writer to a reader through a buffer in the kernel.

A pipe/FIFO works the same whether the writer and reader got connected by making system calls like open("/path/to/named_pipe", O_WRONLY);, or with a pipe(2) to create a new anonymous pipe and return open file descriptors to both the read and write sides.

fstat(2) on the pipe file descriptor will give you sb.st_mode & S_IFMT == S_IFIFO either way.

When you run foo | bar:

  • The shell forks like normal for any non-builtin command
  • Then makes a pipe(2) system call to get two file descriptors: the input and output of an anonymous pipe.
  • Then it forks again.
  • The child (where fork() returned 0)
    • closes the read side of the pipe (leaving the write fd open)
    • and redirects stdout to the write fd with dup2(pipefd[1], 1)
    • then does execve("/usr/bin/foo", ...)
  • The parent (where fork() returned the non-0 child PID)
    • closes the write side of the pipe (leaving the read fd open)
    • and redirects stdin from the read fd with dup2(pipefd[0], 0)
    • then does execve("/usr/bin/bar", ...)

You get into a very similar situation if you run foo > named_pipe & bar < named_pipe.

A named pipe is a rendezvous for processes to establish pipes between each other.

The situation is similar to anonymous tmp files vs. files with names. You can open("/path/to/dir", O_TMPFILE | O_RDWR, S_IRUSR | S_IWUSR); to create a temporary file with no name (O_TMPFILE), as if you'd opened "/path/to/dir/tmpfile" with O_CREAT and then unlinked it, leaving you with a file descriptor to a deleted file.

Using linkat, you can even link that anonymous file into the filesystem, giving it a name, if it was created with O_TMPFILE. (You can't do this on Linux for files that you created with a name then deleted, though.)

Tags:

Pipe

Fifo

Recent Posts