Linux uniq Command Summary with Examples Tutorial

Examples of the Linux uniq Command

Beginner

Video Tutorial

Videos can also be accessed from the Linux Essentials Playlist on YouTube.

Linux uniq Command Summary with Examples (4:07)

Video Script

The Command and Why You Need It

Our thirty-second word, or command to memorize is uniq from our category Text.

uniq allows you to find repeating consecutive lines.

Common Linux uniq Options

-options	description
`--help`	Print help screen
`-c`, `--count`	Count repeating lines
`-s`, `--skip-chars=N`	To access help

Recall from the previous video, we created a sorted text file with repeating lines of text, well here we'll do something worthwhile with that file.

Before we start, it helps to think of commands as mini programs and most follow this structure: command -option(s) argument(s).

The uniq command has 12 options, and arguments include the file you'd like to process and optionally, where you'd like the output.

Like most commands, help is available with double-dash --help. The -c option counts the number of matches and -s allows you to skip a number of characters on the front end of lines.

So why is uniq an important command? Well, we're often faced with messy data sets and uniq allows you to clean them up before, say importing into a database or statistical analysis program. And now you know how to do that.

Demonstration

Okay, the best way to embed this in your memory is by typing in your own terminal window.

Find this on your Mac using a program called Terminal. On Linux use Terminal or Konsole, and currently Microsoft is adding this functionality to Windows.

Here we go, the cat command from video 10 shows us our video32.txt file.

$ cat video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the fifth line This is the first line This is the first line This is the fourth line This is the fourth line This is the second line This is the second line This is the thrid line This is the thrid line

See, we have repeats. Well, like this uniq video32.txt, uniq will show us each repeat.

$ uniq video32.txt # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the first line This is the fourth line This is the second line This is the thrid line $ clear

Okay, next, let's count the repeats using uniq -c video32.txt.

$ uniq -c video32.txt 2 2 # A file for video 5 on less to demonstrate the -N option 2 This is the fifth line 2 This is the first line 2 This is the fourth line 2 This is the second line 2 This is the thrid line

Okay good, two for each, makes sense. Oh and that misspelling (thrid) still bothers me. We'll get that.

And last, what if out front a field made the line unique, but you wanted to ignore it, like a timestamp field for example. I'll spend a couple seconds here editing a few things using the program nano to make the point, like adding a character that would make it unique. And then watch.

$ nano video32.txt

(the file below is before edits)

GNU nano 2.2.6 File: video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the fifth line This is the first line This is the first line This is the fourth line This is the fourth line This is the second line This is the second line This is the thrid line This is the thrid line

(the file below is after edits)

GNU nano 2.2.6 File: video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option aThis is the fifth line bThis is the fifth line cThis is the first line dThis is the first line eThis is the fourth line fThis is the fourth line gThis is the second line hThis is the second line iThis is the thrid line jThis is the thrid line

Now when I use uniq video32.txt and you can see the difference (below).

$ uniq video32.txt # A file for video 5 on less to demonstrate the -N option aThis is the fifth line bThis is the fifth line cThis is the first line dThis is the first line eThis is the fourth line fThis is the fourth line gThis is the second line hThis is the second line iThis is the thrid line jThis is the thrid line

Now, we could use the uniq --skip-chars=1 option to skip the first character.

$ uniq --skip-chars=1 video32.txt # A file for video 5 on less to demonstrate the -N option aThis is the fifth line cThis is the first line eThis is the fourth line gThis is the second line iThis is the thrid line

Interesting huh?

A Final Tip

Okay now you know how to use uniq. And you know the syntax for commands, options and arguments.

One last tip about the uniq command. So uniq helps us make large data sets useful, right? So we'll stick with that theme for the next few videos.

Okay, thanks for visiting today. I hope this was a helpful introduction to the uniq command.

Learn More About The Series

For an overview of 100 videos, the 8 quizzes, a cheat sheet, the categories and a Q&A section, visit:

Linux uniq Command Summary with Examples