FactorPad
Build a Better Process

Linux uniq Command Summary with Examples

Linux uniq allows you to find repeating consecutive lines.
  1. Purpose - Learn what uniq is for and how to find help.
  2. Options - Review a few common options and arguments.
  3. Examples - Walk through code examples with uniq.
  4. A tip - Finish off with one more insight.
face pic by Paul Alan Davis, CFA
Updated: February 23, 2021
In this tutorial on Linux uniq, 32 of 100, below find a 3-4 minute introductory video, a text-based tutorial and all of the code examples from the video.

Outline Back Tip Next

/ factorpad.com / tech / linux-essentials / uniq-command.html


An ad-free and cookie-free website.


Examples of the Linux uniq Command

Beginner

Video Tutorial

Videos can also be accessed from the Linux Essentials Playlist on YouTube.

Linux uniq Command Summary with Examples (4:07)

Video Script

The Command and Why You Need It

Our thirty-second word, or command to memorize is uniq from our category Text.

uniq allows you to find repeating consecutive lines.

Common Linux uniq Options
-options description
--help Print help screen
-c, --count Count repeating lines
-s, --skip-chars=N To access help

Recall from the previous video, we created a sorted text file with repeating lines of text, well here we'll do something worthwhile with that file.

Before we start, it helps to think of commands as mini programs and most follow this structure: command -option(s) argument(s).

The uniq command has 12 options, and arguments include the file you'd like to process and optionally, where you'd like the output.

Like most commands, help is available with double-dash --help. The -c option counts the number of matches and -s allows you to skip a number of characters on the front end of lines.

So why is uniq an important command? Well, we're often faced with messy data sets and uniq allows you to clean them up before, say importing into a database or statistical analysis program. And now you know how to do that.

Demonstration

Okay, the best way to embed this in your memory is by typing in your own terminal window.

Find this on your Mac using a program called Terminal. On Linux use Terminal or Konsole, and currently Microsoft is adding this functionality to Windows.

Here we go, the cat command from video 10 shows us our video32.txt file.

$ cat video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the fifth line This is the first line This is the first line This is the fourth line This is the fourth line This is the second line This is the second line This is the thrid line This is the thrid line

See, we have repeats. Well, like this uniq video32.txt, uniq will show us each repeat.

$ uniq video32.txt # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the first line This is the fourth line This is the second line This is the thrid line $ clear

Okay, next, let's count the repeats using uniq -c video32.txt.

$ uniq -c video32.txt 2 2 # A file for video 5 on less to demonstrate the -N option 2 This is the fifth line 2 This is the first line 2 This is the fourth line 2 This is the second line 2 This is the thrid line

Okay good, two for each, makes sense. Oh and that misspelling (thrid) still bothers me. We'll get that.

And last, what if out front a field made the line unique, but you wanted to ignore it, like a timestamp field for example. I'll spend a couple seconds here editing a few things using the program nano to make the point, like adding a character that would make it unique. And then watch.

$ nano video32.txt

(the file below is before edits)

GNU nano 2.2.6 File: video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option This is the fifth line This is the fifth line This is the first line This is the first line This is the fourth line This is the fourth line This is the second line This is the second line This is the thrid line This is the thrid line

(the file below is after edits)

GNU nano 2.2.6 File: video32.txt # A file for video 5 on less to demonstrate the -N option # A file for video 5 on less to demonstrate the -N option aThis is the fifth line bThis is the fifth line cThis is the first line dThis is the first line eThis is the fourth line fThis is the fourth line gThis is the second line hThis is the second line iThis is the thrid line jThis is the thrid line

Now when I use uniq video32.txt and you can see the difference (below).

$ uniq video32.txt # A file for video 5 on less to demonstrate the -N option aThis is the fifth line bThis is the fifth line cThis is the first line dThis is the first line eThis is the fourth line fThis is the fourth line gThis is the second line hThis is the second line iThis is the thrid line jThis is the thrid line

Now, we could use the uniq --skip-chars=1 option to skip the first character.

$ uniq --skip-chars=1 video32.txt # A file for video 5 on less to demonstrate the -N option aThis is the fifth line cThis is the first line eThis is the fourth line gThis is the second line iThis is the thrid line

Interesting huh?

A Final Tip

Okay now you know how to use uniq. And you know the syntax for commands, options and arguments.

One last tip about the uniq command. So uniq helps us make large data sets useful, right? So we'll stick with that theme for the next few videos.

Okay, thanks for visiting today. I hope this was a helpful introduction to the uniq command.


Learn More About The Series

For an overview of 100 videos, the 8 quizzes, a cheat sheet, the categories and a Q&A section, visit:


What's Next?

If you haven't paired text-based tutorials with our videos it may help you learn faster. Subscribe and check out the YouTube Channel for notifications of new content.

Outline Back Tip Next

/ factorpad.com / tech / linux-essentials / uniq-command.html


linux uniq command
uniq linux
linux uniq
unix uniq
uniq unix
linux find repeat
uniq command examples
remove repeats
man uniq
uniq command syntax
linux uniq summary
uniq command options
uniq command tutorial
uniq examples
remove duplicate lines
shell uniq
linux remove duplicates

A newly-updated free resource. Connect and refer a friend today.