Updated 2017-08-01 to fix typo and use new demo URL. Also note that the Homebrew package manager has matured since this post was first published in 2012. Once you’ve configured Homebrew, installing a tool like wget is as simple as runningbrew install wget
wget is an incredibly useful GNU tool on Linux. Unfortunately, it doesn’t come with OS X (as of Mountain Lion). OS X includes curl, which is a very handy tool but lacks at least one important feature of wget: the ability to use wildcards to get multiple files at the same time. For example, let’s say you want to download a subset of files from an FTP server. With wget, you could type:
Here is how to mimic that process with curl and a few UNIX command-line tricks.
1. Download the directory listing and save it in a file.
Curl is another effective download tool, it can be use to upload or download file with giving a simple command, it supports pause or resume of downloaded package and supports maximum of web protocols, it can predict time left in download to be complete, progress is visible via progress bar. Download@Authors Site Bad Link Rating: 1 (1 votes) 1.Bad 2.Not Geeky 3.Average 4.Good 5.Geek-o-licious curl is an open-source command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS and FILE.
2. Use grep with regular expressions to parse the .html file, extract the .igs file names and save them in a text file.
3. Use a bash loop to iterate over the text file and fetch each file with curl.
Another advantage of the curl approach is that wget doesn’t support wildcard characters with HTTP URLs (only FTP).
Question: I typically use wget to download files. On some systems, wget is not installed and only curl is available. Can you explain me with a simple example on how I can download a remote file using curl? Are there any difference between curl and wget?
Answer: On a high-level, both wget and curl are command line utilities that do the same thing.
- They both can be used to download files using FTP and HTTP(s).
- You can also send HTTP POST request using curl and wget
- However curl provides APIs that can be used by programmers inside their own code. curl uses libcurl which is a cross-platform library.
- wget is just a command-line tool without any APIs.
- Curl also supports lot more protocols that wget doesn’t support. For example: SCP, SFTP, TFTP, TELNET, LDAP(S), FILE, POP3, IMAP, SMTP, RTMP and RTSP.
- There is a major advantage of using wget. wget supports recursive download, while curl doesn’t.
Wget Examples
The following example downloads the file and stores in the same name as the remote server.
The following example download the file and stores in a different name than the remote server. This is helpful when the remote URL doesn’t contain the file name in the url as shown in the example below.
Curl Download File
More wget examples: The Ultimate Wget Download Guide With 15 Awesome Examples
Curl Examples
Option -O (upper-case O) is important. Without this, curl will start dumping the downloaded file on the stdout. Using -O, it downloads the files in the same name as the remote server. In the above example, we are downloading strx25-0.9.2.1.tar.bz2, so the downloaded file will also be the same name.
Instead of -O, you an also specify, “–remote-name” as shown below. Both are the same.
While curl is downloading it gives the following useful information:
- % – The total % of the download that was completed as of now. When it gets to 100% the download is completed. In the above example, it has downloaded only 28% of the file.
- Total – The total size of the file
- Received – The total size of the file that was has been downloaded so far. In the above example, it has downloaded 1085k so far (out of 3762k total)
- Xferd – This will be used when you upload some files to the remote server. During upload, this will indicate the total size of the file that has been uploaded so far. Since we are downloading a file, in this example, this is 0.
- Average Speed Dload – This indicates the average download speed.
- AVerage Speed Upload – While uploading a file, this will indicate the average upload speed
- Time Total – This indicates the total time it will take to download (or upload) the whole file based on the current download (or upload) speed. In this example, it will take approximately a total of 52 seconds to download this file.
- Time Spend – The time curl has spent so far downloading (or uploading) the file. In this example, it has spent 15 seconds so far.
- Time Left – This is caculated based on “Time Total” – “Time Spent”.
- Current Speed – This indicates the current download/upload speed. Compare this with Average Spped Dload/UPload to see how fast or slow your system is downloading currently.
If you want to download the file and store it in a different name than the name of the file in the remote server, use -o (lower-case o) as shown below. This is helpful when the remote URL doesn’t contain the file name in the url as shown in the example below.
In the above example, there is no file name in the remote URL, it just calls a php script that passes some parameter to it. However, the file will be downloaded and saved as taglist.zip on your local system. Instead of -o, you an also specify, “–output”.
More curl examples: 15 Practical Linux cURL Command Examples
If you enjoyed this article, you might also like..
Next post: File Handling in C with Examples (fopen, fread, fwrite, fseek)
Windows Curl Download File
Previous post: What is Linux System Calls and Library Functions?