How to generate list of unique lines in text file using a Linux shell script?
cat
to output the contents, piped to sort
to sort them, piped to uniq
to print out the unique values:
cat test1.txt | sort | uniq
you don't need to do the sort
part if the file contents are already sorted.
If you don't mind the output being sorted, use
sort -u
This sorts and removes duplicates