Hack 24. Uniq Command Examples

by Ramesh

Uniq command is mostly used in combination with sort command, as uniq removes duplicates only from a sorted file. i.e In order for uniq to work, all the duplicate entries should be in the adjacent lines. Following are some common examples.

1. Remove Duplicates from a file using uniq command

When you have an employee file with duplicate entries, you can do the following to remove duplicates.

$ sort namesd.txt | uniq
 
$ sort –u namesd.txt

2. Display Duplicate Row Count Using Uniq

If you want to know how many lines are duplicates, do the following. The first field in the following examples indicates how many duplicates where found for that particular line. So, in this example the lines beginning with Alex and Emma were found twice in the namesd.txt file.

$ sort namesd.txt | uniq –c
 
      2 Alex Jason:200:Sales
      2 Emma Thomas:100:Marketing
      1 Madison Randy:300:Product Development
      1 Nisha Singh:500:Sales
      1 Sanjay Gupta:400:Support

3. Display Only Duplicate Rows Using Uniq

The following displays only the entries that are duplicates.

$ sort namesd.txt | uniq –cd
 
      2 Alex Jason:200:Sales
      2 Emma Thomas:100:Marketing