Bash history: "ignoredups" and "erasedups" setting conflict with common history across sessions

This is actually a really interesting behavior and I confess I have greatly underestimated the question at the beginning. But first the facts:

###1. What works

The functionality can be achieved in several ways, though each works a bit differently. Note that, in each case, to have the history "transferred" to another terminal (updated), one has to press Enter in the terminal, where he/she wants to retrieve the history.

  • option 1:

     shopt -s histappend
 HISTCONTROL=ignoredups
 PROMPT_COMMAND="history -a; history -n; $PROMPT_COMMAND"

This has two drawbacks:

  1. At login (opening a terminal), the last command from the history file is read twice into the current terminal's history buffer;
  2. The buffers of different terminals do not stay in sync with the history file.

  • option 2:

     HISTCONTROL=ignoredups
 PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"

(Yes, no need for shopt -s histappend and yes, it has to be history -c in the middle of PROMPT_COMMAND) This version has also two important drawbacks:

  1. The history file has to be initialized. It has to contain at least one non-empty line (can be anything).
  2. The history command can give false output - see below.

[Edit] "And the winner is..."

  • option 3:

     HISTCONTROL=ignoredups:erasedups
 shopt -s histappend
 PROMPT_COMMAND="history -n; history -w; history -c; history -r; $PROMPT_COMMAND"

This is as far as it gets. It is the only option to have both erasedups and common history working simultaneously. This is probably the final solution to all your problems, Aahan.

###2. Why does option 2 not seem to work (or: what really doesn't work as expected)? As I mentioned, each of the above solutions works differently. But the most misleading interpretation of how the settings work comes from analysing the output of history command. In many cases, the command can give false output. Why? Because it is executed before the sequence of other history commands contained in the PROMPT_COMMAND! However, when using the second or third option, one can monitor the changes of .bash_history contents (using watch -n1 "tail -n20 .bash_history" for example) and see what the real history is.

###3. Why option 3 is so complicated? It all lies in the way erasedups works. As the bash manual states, "(...) erasedups causes all previous lines matching the current line to be removed from the history list before that line is saved". So this is really what the OP wanted (and not just, as I previously thought, to have no duplicates appearing in sequence). Here's why each of the history -. commands either has to or can not be in the PROMPT_COMMAND:

  • history -n has to be there before history -w to read from .bash_history the commands saved from any other terminal,

  • history -w has to be there in order to save the new history to the file after bash has checked if the command was a duplicate,

  • history -a must not be placed there instead of history -w, because it will add to the file any new command, regardless of whether it was checked as a duplicate.

  • history -c is also needed because it prevents trashing the history buffer after each command,

  • and finally, history -r is needed to restore the history buffer from file, thus finally making the history shared across terminal sessions.

Be aware that this solution will mess up the history order by putting all history from other terminals in front of the newest command entered in the current terminal. It also does not delete duplicate lines already in the history file unless you enter that command again.


In your prompt command, you are using the -c switch. From man bash:

-c   Clear the history list by deleting all the entries

To share your history with all open terminals, you can use -n:

-n   Read the history lines not already read from the history file into the current history list. These are lines appended to the history file since the beginning of the current bash session.

The default size is also in the manual:

HISTSIZE The number of commands to remember in the command history (see HISTORY below). The default value is 500.

To save multi-line commands:

The cmdhist shell option, if enabled, causes the shell to attempt to save each line of a multi-line command in the same history entry, adding semicolons where necessary to preserve syntactic correctness. The lithist shell option causes the shell to save the command with embedded newlines instead of semicolons.

Also, you shouldn't preface HIST* commands with export — they are bash-only variables not environmental variables: HISTCONTROL=ignoredups:erasedups is sufficient.


This is what I came up with and I am happy with it so far…

alias hfix='history -n && history | sort -k2 -k1nr | uniq -f1 | sort -n | cut -c8- > ~/.tmp$$ && history -c && history -r ~/.tmp$$ && history -w && rm ~/.tmp$$'  
HISTCONTROL=ignorespace  
shopt -s histappend  
shopt -s extglob  
HISTSIZE=1000  
HISTFILESIZE=2000  
export HISTIGNORE="!(+(*\ *))"  
PROMPT_COMMAND="hfix; $PROMPT_COMMAND"

NOTES:

  • Yes, it is complicated... but, it removes all duplicates and yet preserves chronology within each terminal!
  • My HISTIGNORE ignores all commands that don't have arguments. This may not be desirable by some folks and can be left out.

Tags:

Bash

Command History

Recent Posts