Printing unique lines
To print each identical line only one, in any order:
sort -u
To print only the unique lines, in any order:
sort | uniq -u
To print each identical line only once, in the order of their first occurrence: (for each line, print the line if it hasn't been seen yet, then in any case increment the seen counter)
awk '!seen[$0] {print}
{++seen[$0]}'
To print only the unique lines, in the order of their first occurrence: (record each line in seen
, and also in lines
if it's the first occurrence; at the end of the input, print the lines in order of occurrence but only the ones seen only once)
awk '!seen[$0]++ {lines[i++]=$0}
END {for (i in lines) if (seen[lines[i]]==1) print lines[i]}'
Some (most?) versions of sort
have a -u
flag that does the uniq
part directly. Might be some line length restrictions depending on the implementation though, but you had those already with plain sort|uniq
.