FactorPad
Faster Learning Tutorials

Linux wget command summary with examples

wget allows you to download files non-interactively.
  1. Purpose - Learn what wget is for and how to find help.
  2. Options - Review a few common options and arguments.
  3. Examples - Walk through code examples with wget.
  4. A tip - Finish off with one more insight.
Paul Alan Davis, CFA, November 12, 2016
Updated: July 25, 2018
In this tutorial, 85 of 100, below find a 3-4 minute introductory video, a text-based tutorial and all of the code examples from the video.

Outline Back Next

~/ home  / technology  / linux essentials  / wget command


The Linux wget command

Beginner

Learn to download files at the Linux command line.

Video Tutorial

Linux wget command summary with examples (3:40)

Videos can also be accessed from the YouTube Playlist.

Video Script

The Command and Why You Need It

Our eighty-fifth word, or command to memorize is wget from our category Communication.

wget allows you to download files non-interactively.

Recall from the last video (tutorial) on ftp we manually logged in to a remote site, and now we'll download files non-interactively.

Before we start, it helps to think of commands as mini programs and most follow this structure: command -option(s) argument(s).

The wget command has over 130 options, and yes, I counted. (Please review man wget.) The argument is the URL, or Uniform Resource Locator, which is that long field at the top of your browser pointing to a file using a variety of Protocols, normally http, but could also be https, ftp and others.

Like most commands, help is available with double-dash --help. The -b option will kick time-intensive operations into the background, and -a will append messages to a specified log file.

So why is wget an important command? Well, organizations, like search engines, need an automated way to scrape and process information from the Internet. And now you know how to do that.

Demonstration

Okay, the best way to embed this in your memory is by typing in your own terminal window.

Find this on your Mac using a program called Terminal. On Linux use Terminal or Konsole, and currently Microsoft is adding this functionality to Windows.

Here we go. So wget can get very complex, but let's try it for something simple, and something I've been curious about. Let's use less to see if that simple-looking Google homepage is simple.

$ wget google.com -O video85a.txt; less video85a.txt --2016-11-12 20:18:02-- http://google.com/ Resolving google.com (google.com)... 64.233.191.138, 64.233.191.101, 64.233.191.100, ... Connecting to google.com (google.com)|64.233.191.138|:80... connected. HTTP request sent, awaiting response.. 301 Moved Permanently Location: http://www.google.com/ [following] --2016-11-12 20:18:02-- http://www.google.com/ Resolving www.google.com (www.google.com)... 74.125.193.106, 74.125.193.105, 74.125.193.104, ... Connecting to www.google.com (www.google.com)|74.125.193.106|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: 'video85a.txt' video85a.txt [ <=> ] 10.05K --.-KB/s in 0s 2016-11-12 20:18:02 (165 MB/s) - 'video85a.txt' saved [10288]

(The output below is a subset of the html file downloaded from the google.com home page.)

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en"><head><meta content="Search t he world's information, including webpages, images, videos and more. Google has many special features to help you find exactly what you're looking for." name="description"><meta content="noodp name="robots"><meta conten t="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_stan dard_color_128dp.png" itemprop="image"><title><Google</title><script>(function(){window.google={kEI:'euknWL2AOo SpjWTLu5_ADw',kEXPI:'1351827,1351903,1352084,3700323,402641,4029815,4031109,4032678,4036527,4038012,4039268,4 043492,4045841,4048347,4052304,4053261,4055745,4065787,4066688,4068550,4069838,4069840,4070138,4072364,4072602 ,4072705,4072773,4073111,4073405,4073418,4073757,4073915,4073959,4079597,4079626,4080167,4080383,4080535,408 0639,4080640,4080815,4081037,4081038,4081128,4081264,4081470,4081913,4082309,4082939,4082942,4 (many lines trimmed, you get the point)

Okay, well looking at it, it doesn't seem as complex as you might think (sarcasm intended), although the formatting is kind of strange here (no sarcasm intended here). ;)

And next, there's another public file I know something about, it's a data set I use for teaching an investment modeling boot camp on YouTube, the file is at factorpad.com/returns.txt.

$ wget factorpad.com/returns.txt -O video85b.txt --2016-11-12 20:18:30-- http://factorpad.com/returns.txt Resolving factorpad.com (factorpad.com)...69.195.124.226 Connecting to factorpad.com (factorpad.com)|69.195.124.226|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4080 (4.0K) [text/plain] Saving to: 'video85b.txt' video85b.txt 100%[==========================================>] 3.98K --.-KB/s in 0s 2016-11-12 20:18:30 (548 MB/s) - 'video85b.txt' saved [4080/4080]

And then last, let's review it using head.

$ head video85b.txt Date MSFT EBAY ABT MRK Market Tbill 4/30/2003 0.056175 0.089087 0.083536 0.062066 0.072716 0.001000 5/30/2003 -0.037544 0.094070 0.096480 -0.044689 0.027079 0.000900 6/30/2003 0.041853 0.023119 -0.017733 0.096319 0.035890 0.001000 7/31/2003 0.030031 0.031731 -0.100672 -0.087036 -0.031487 0.000700 8/29/2003 0.004165 0.032805 0.026752 -0.039496 0.006057 0.000700 9/30/2003 0.048265 -0.031944 0.055831 0.013306 0.021365 0.000800 10/31/2003 -0.054537 0.042692 0.004456 -0.125840 -0.033307 0.000700 11/28/2003 -0.016450 -0.000715 0.037072 -0.082486 -0.015645 0.000700 12/31/2003 0.064566 0.156021 0.054299 0.147581 0.105617 0.000800

It has returns on four stocks, an Index and Treasury bills for 5-years and we use that to geek out on Statistics in Finance (Quant 101). Check that out if you are interested.

A Final Tip

Okay now you know how to use wget. And you know the syntax for commands, options and arguments.

One last tip about the wget command. So wget can also take a long list of files as input, it can follow links recursively and it'll even eat cookies.

Okay, thanks for visiting today. I hope this was a helpful introduction to the wget command.


Learn More About The Series

For an overview of the 100 videos, the 8 quizzes, a cheat sheet, the categories and a Q&A section, visit:


What's Next?

Our YouTube Channel is growing, subscribe here to be a part of it.

  • For the Outline to all 100 tutorials, click Outline.
  • To go back to the ftp command, hit Back.
  • To learn about the ssh command, click Next.

Outline Back Next

~/ home  / tech  / linux essentials  / wget command



 
 
Keywords:
Linux wget command
wget example
networking commands
download full website
man wget
wget recursive
wget directory
download files
download cookies
wget http
wget ssl
download list of files
scrape websites
search engines