uniq Command

uniq removes duplicate consecutive lines in a text file and is useful for simplifying the text display.

Because uniq requires that the duplicate entries must be consecutive, one often runs sort first and then pipes the output into uniq; if sort is used with the -u option, it can do all this in one step.

To remove duplicate entries from multiple files at once, use the following command:

1
sort file1 file2 | uniq > file3

or

1
sort -u file1 file2 > file3

To count the number of duplicate entries, use the following command:

1
uniq -c filename