Linux wget Command Summary with Examples Tutorial

Examples of the Linux wget Command

Beginner

Learn to download files at the Linux command line.

Video Tutorial

Videos can also be accessed from the Linux Essentials Playlist on YouTube.

Linux wget Command Summary with Examples (3:40)

Video Script

The Command and Why You Need It

Our eighty-fifth word, or command to memorize is wget from our category Communication.

wget allows you to download files non-interactively.

Common Linux wget Options

-options	description
`-h`, `--help`	Print help screen
`-b`, `--background`	Operate in the background
`-a`, `--append-output=FILE`	Append messages to FILE

Recall from the last video (tutorial) on ftp we manually logged in to a remote site, and now we'll download files non-interactively.

Before we start, it helps to think of commands as mini programs and most follow this structure: command -option(s) argument(s).

The wget command has over 130 options, and yes, I counted. (Please review man wget.) The argument is the URL, or Uniform Resource Locator, which is that long field at the top of your browser pointing to a file using a variety of Protocols, normally http, but could also be https, ftp and others.

Like most commands, help is available with double-dash --help. The -b option will kick time-intensive operations into the background, and -a will append messages to a specified log file.

So why is wget an important command? Well, organizations, like search engines, need an automated way to scrape and process information from the Internet. And now you know how to do that.

Demonstration

Okay, the best way to embed this in your memory is by typing in your own terminal window.

Find this on your Mac using a program called Terminal. On Linux use Terminal or Konsole, and currently Microsoft is adding this functionality to Windows.

Here we go. So wget can get very complex, but let's try it for something simple, and something I've been curious about. Let's use less to see if that simple-looking Google homepage is simple.

$ wget google.com -O video85a.txt; less video85a.txt --2016-11-12 20:18:02-- http://google.com/ Resolving google.com (google.com)... 64.233.191.138, 64.233.191.101, 64.233.191.100, ... Connecting to google.com (google.com)|64.233.191.138|:80... connected. HTTP request sent, awaiting response.. 301 Moved Permanently Location: http://www.google.com/ [following] --2016-11-12 20:18:02-- http://www.google.com/ Resolving www.google.com (www.google.com)... 74.125.193.106, 74.125.193.105, 74.125.193.104, ... Connecting to www.google.com (www.google.com)|74.125.193.106|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: 'video85a.txt' video85a.txt [ <=> ] 10.05K --.-KB/s in 0s 2016-11-12 20:18:02 (165 MB/s) - 'video85a.txt' saved [10288]

(The output below is a subset of the html file downloaded from the google.com home page.)

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search t he world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp name="robots"><meta conten t="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_stan dard_color_128dp.png" itemprop="image"><title><Google</title><script>(function(){window.google={kEI:'euknWL2AOo SpjWTLu5_ADw',kEXPI:'1351827,1351903,1352084,3700323,402641,4029815,4031109,4032678,4036527,4038012,4039268,4 043492,4045841,4048347,4052304,4053261,4055745,4065787,4066688,4068550,4069838,4069840,4070138,4072364,4072602 ,4072705,4072773,4073111,4073405,4073418,4073757,4073915,4073959,4079597,4079626,4080167,4080383,4080535,408 0639,4080640,4080815,4081037,4081038,4081128,4081264,4081470,4081913,4082309,4082939,4082942,4 (many lines trimmed, you get the point)

Okay, well looking at it, it doesn't seem as complex as you might think (sarcasm intended), although the formatting is kind of strange here (no sarcasm intended here). ;)

And next, there's another public file I know something about, it's a data set I use for teaching an investment modeling boot camp on YouTube, the file is at factorpad.com/returns.txt.

$ wget factorpad.com/returns.txt -O video85b.txt --2016-11-12 20:18:30-- http://factorpad.com/returns.txt Resolving factorpad.com (factorpad.com)...69.195.124.226 Connecting to factorpad.com (factorpad.com)|69.195.124.226|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4080 (4.0K) [text/plain] Saving to: 'video85b.txt' video85b.txt 100%[==========================================>] 3.98K --.-KB/s in 0s 2016-11-12 20:18:30 (548 MB/s) - 'video85b.txt' saved [4080/4080]

And then last, let's review it using head.

$ head video85b.txt Date MSFT EBAY ABT MRK Market Tbill 4/30/2003 0.056175 0.089087 0.083536 0.062066 0.072716 0.001000 5/30/2003 -0.037544 0.094070 0.096480 -0.044689 0.027079 0.000900 6/30/2003 0.041853 0.023119 -0.017733 0.096319 0.035890 0.001000 7/31/2003 0.030031 0.031731 -0.100672 -0.087036 -0.031487 0.000700 8/29/2003 0.004165 0.032805 0.026752 -0.039496 0.006057 0.000700 9/30/2003 0.048265 -0.031944 0.055831 0.013306 0.021365 0.000800 10/31/2003 -0.054537 0.042692 0.004456 -0.125840 -0.033307 0.000700 11/28/2003 -0.016450 -0.000715 0.037072 -0.082486 -0.015645 0.000700 12/31/2003 0.064566 0.156021 0.054299 0.147581 0.105617 0.000800

It has returns on four stocks, an Index and Treasury bills for 5-years and we use that to geek out on Statistics in Finance (Quant 101). Check that out if you are interested.

A Final Tip

Okay now you know how to use wget. And you know the syntax for commands, options and arguments.

One last tip about the wget command. So wget can also take a long list of files as input, it can follow links recursively and it'll even eat cookies.

Okay, thanks for visiting today. I hope this was a helpful introduction to the wget command.

Learn More About The Series

For an overview of the 100 videos, the 8 quizzes, a cheat sheet, the categories and a Q&A section, visit:

Linux wget Command Summary with Examples