Linux: How to use a file as input and output at the same time?

Solution 1:

echo "$(uniq .bash_history)" > .bash_history

should have the desired result. The subshell gets executed before .bash_history is opened for writing. As explained in Phil P's answer, by the time .bash_history is read in the original command, it has already been truncated by the > operator.

Solution 2:

I recommend using sponge from moreutils. From the manpage:

DESCRIPTION
  sponge  reads  standard  input  and writes it out to the specified file. Unlike
  a shell redirect, sponge soaks up all its input before opening the output file.
  This allows for constructing pipelines that read from and write to the same 
  file.

To apply this to your problem, try:

uniq .bash_history | sponge .bash_history

---

Solution 3:

The problem is that your shell is setting up the command pipeline before running the commands. It's not a matter of "input and output", it's that the file's content is already gone before uniq even runs. It goes something like:

1. The shell opens the > output file for writing, truncating it
2. The shell sets up to have file-descriptor 1 (for stdout) be used for that output
3. The shell executes uniq, perhaps something like execlp("uniq", "uniq", ".bash_history", NULL)
4. uniq runs, opens .bash_history and finds nothing there

There are various solutions, including the in-place editing and the temporary file usage which others mention, but the key is to understand the problem, what's actually going wrong and why.

---

Solution 4:

Another trick to do this, without using sponge, is the following command:

{ rm .bash_history && uniq > .bash_history; } < .bash_history

This is one of the cheats described in the excellent article “In-place” editing of files on backreference.org.

It basically opens the file for reading, then "removes" it. It's not really removed, though: There's an open file descriptor pointing to it, and as long as that remains open, the file is still around. Then it creates a new file with the same name and writes the unique lines to it.

Disadvantage of this solution: If uniq fails for some reason, your history will be gone.

---

Solution 5:

use sponge from moreutils

uniq .bash_history | sponge .bash_history