FactorPad
Faster Learning Tutorials

Linux uniq command summary with examples

uniq allows you to find repeating consecutive lines.
  1. Purpose - Learn what uniq is for and how to find help.
  2. Options - Review a few common options and arguments.
  3. Examples - Walk through code examples with uniq.
  4. A tip - Finish off with one more insight.
Paul Alan Davis, CFA, October 18, 2016
Updated: July 31, 2018
In this tutorial, 32 of 100, below find a 3-4 minute introductory video, a text-based tutorial and all of the code examples from the video.

Outline Back Next

~/ home  / tech  / linux essentials  / uniq command


The Linux uniq command

Beginner

Video Tutorial

Linux uniq command summary with examples (4:07)

Videos can also be accessed from the YouTube Playlist.

Video Script

The Command and Why You Need It

Our thirty-second word, or command to memorize is uniq from our category Text.

uniq allows you to find repeating consecutive lines.

Recall from the previous video, we created a sorted text file with repeating lines of text, well here we'll do something worthwhile with that file.

Before we start, it helps to think of commands as mini programs and most follow this structure: command -option(s) argument(s).

The uniq command has 12 options, and arguments include the file you'd like to process and optionally, where you'd like the output.

Like most commands, help is available with double-dash --help. The -c option counts the number of matches and -s allows you to skip a number of characters on the front end of lines.

So why is uniq an important command? Well, we're often faced with messy data sets and uniq allows you to clean them up before, say importing into a database or statistical analysis program. And now you know how to do that.

Demonstration

Okay, the best way to embed this in your memory is by typing in your own terminal window.

Find this on your Mac using a program called Terminal. On Linux use Terminal or Konsole, and currently Microsoft is adding this functionality to Windows.

Here we go, the cat command from video 10 shows us our video32.txt file.

$ cat video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the fifth line This is the first line This is the first line This is the fourth line This is the fourth line This is the second line This is the second line This is the thrid line This is the thrid line

See, we have repeats. Well, like this uniq video32.txt, uniq will show us each repeat.

$ uniq video32.txt # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the first line This is the fourth line This is the second line This is the thrid line $ clear

Okay, next, let's count the repeats using uniq -c video32.txt.

$ uniq -c video32.txt 2 2 # A file for video 5 on less to demonstrate the -N option 2 This is the fifth line 2 This is the first line 2 This is the fourth line 2 This is the second line 2 This is the thrid line

Okay good, two for each, makes sense. Oh and that misspelling (thrid) still bothers me. We'll get that.

And last, what if out front a field made the line unique, but you wanted to ignore it, like a timestamp field for example. I'll spend a couple seconds here editing a few things using the program nano to make the point, like adding a character that would make it unique. And then watch.

$ nano video32.txt

(the file below is before edits)

GNU nano 2.2.6 File: video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the fifth line This is the first line This is the first line This is the fourth line This is the fourth line This is the second line This is the second line This is the thrid line This is the thrid line

(the file below is after edits)

GNU nano 2.2.6 File: video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option aThis is the fifth line bThis is the fifth line cThis is the first line dThis is the first line eThis is the fourth line fThis is the fourth line gThis is the second line hThis is the second line iThis is the thrid line jThis is the thrid line

Now when I use uniq video32.txt and you can see the difference (below).

$ uniq video32.txt # A file for video 5 on less to demonstrate the -N option aThis is the fifth line bThis is the fifth line cThis is the first line dThis is the first line eThis is the fourth line fThis is the fourth line gThis is the second line hThis is the second line iThis is the thrid line jThis is the thrid line

Now, we could use the uniq --skip-chars=1 option to skip the first character.

$ uniq --skip-chars=1 video32.txt # A file for video 5 on less to demonstrate the -N option aThis is the fifth line cThis is the first line eThis is the fourth line gThis is the second line iThis is the thrid line

Interesting huh?

A Final Tip

Okay now you know how to use uniq. And you know the syntax for commands, options and arguments.

One last tip about the uniq command. So uniq helps us make large data sets useful, right? So we'll stick with that theme for the next few videos.

Okay, thanks for visiting today. I hope this was a helpful introduction to the uniq command.


Learn More About The Series

For an overview of 100 videos, the 8 quizzes, a cheat sheet, the categories and a Q&A section, visit:


What's Next?

If you haven't paired text-based tutorials with our videos it may help you learn faster. Subscribe and check out the YouTube Channel.

  • For the Outline to all 100 tutorials, click Outline.
  • To go back to the sort command, hit Back.
  • To continue this exercise with the cut command, click Next.

Outline Back Next

~/ home  / tech  / linux essentials  / uniq command



 
 
Keywords:
linux uniq command
uniq linux
unix uniq
linux find repeats
linux remove repeats
linux count repeats
linux repeated lines
man uniq
uniq command syntax
linux uniq summary
uniq command options
uniq command tutorial
linux remove duplicates
shell uniq
remove duplicate lines