Check all lines of a file are unique
Awk solution:
awk 'a[$0]++{print "dupes"; exit(1)}' file && echo "no dupes"
[ "$(wc -l < input)" -eq "$(sort -u input | wc -l)" ] && echo all unique
Using sort
/uniq
:
sort input.txt | uniq
To check only for duplicate lines use the -d
option for uniq. This will show only lines that are duplicate, if none it will show nothing:
sort input.txt | uniq -d