How to create an array of unique elements from a string/array in bash?

If you are using zsh:

$ array=(1 2 3 2 1)
$ echo ${(u)array[@]}
1 2 3

or (if KSH_ARRAYS option is not set) even

$ echo ${(u)array}
1 2 3

With GNU awk (this also retains original order)

printf '%s\n' "1 2 3 2 1" | awk -v RS='[[:space:]]+' '!a[$0]++{printf "%s%s", $0, RT}'
1 2 3

To read into a bash array

read -ra arr<<<$(printf '%s\n' "1 2 3 2 1" |
 awk -v RS='[[:space:]]+' '!a[$0]++{printf "%s%s", $0, RT}')
printf "%s\n"  "${arr[@]}"
1
2
3

For an array with arbitrary values, it's quite tricky with bash as it doesn't have a builtin operator for that.

bash however happens not to support storing NUL characters in its variables, so you can make use of that to pass that to other commands:

The equivalent of zsh's:

new_array=("${(@u}array}")

on a recent GNU system, could be:

eval "new_array=($(
  printf "%s\0" "${array[@]}" |
    LC_ALL=C sort -zu |
    xargs -r0 bash -c 'printf "%q\n" "$@"' sh
  ))"

Alternatively, with recent versions of bash, and assuming none of the array elements are empty, you could use associative arrays:

unset hash
typeset -A hash
for i in "${array[@]}"; do
  hash[$i]=
done
new_array=("${!hash[@]}")

With bash 4.4 and newer and with GNU sort:

readarray -td '' new_array < <(
  printf '%s\0' "${array[@]}" | LC_ALL=C sort -zu)

The order of the elements would not be the same in those different solutions.

With tcsh:

set -f new_array = ($array:q)

Would retain the first element (a b a => a b) like zsh's (u) expansion flag.

set -l new_array = ($array:q)

Would retain the last (a b a => b a). Those however remove empty elements from the array.

Related