How can I remove duplicates in my .bash_history, preserving order?

So I was looking for the same exact thing after being annoyed by duplicates, and found that if I edit my ~/.bash_profile or my ~/.bashrc with:

export HISTCONTROL=ignoreboth:erasedups

It does exactly what you wanted, it only keeps the latest of any command. ignoreboth is actually just like doing ignorespace:ignoredups and that along with erasedups gets the job done.

At least on my Mac terminal with bash this work perfect. Found it here on askubuntu.com.

Sorting the history

This command works like sort|uniq, but keeps the lines in place

nl|sort -k 2|uniq -f 1|sort -n|cut -f 2

Basically, prepends to each line its number. After sort|uniq-ing, all lines are sorted back according to their original order (using the line number field) and the line number field is removed from the lines.

This solution has the flaw that it is undefined which representative of a class of equal lines will make it in the output and therefore its position in the final output is undefined. However, if the latest representative should be chosen you can sort the input by a second key:

nl|sort -k2 -k 1,1nr|uniq -f1|sort -n|cut -f2

Managing .bash_history

For re-reading and writing back the history, you can use history -a and history -w respectively.

Found this solution in the wild and tested:

awk '!x[$0]++'

The first time a specific value of a line ($0) is seen, the value of x[$0] is zero.
The value of zero is inverted with ! and becomes one.
An statement that evaluates to one causes the default action, which is print.

Therefore, the first time an specific $0 is seen, it is printed.

Every next time (the repeats) the value of x[$0] has been incrented,
its negated value is zero, and a statement that evaluates to zero doesn't print.

To keep the last repeated value, reverse the history and use the same awk:

awk '!x[$0]++' ~/.bash_history                 # keep the first value repeated.

tac ~/.bash_history | awk '!x[$0]++' | tac     # keep the last.