In this tutorial, we will introduce the cURL command in Linux. We will give some examples to guide you through the functions of this powerful utility and help you understand all the functions it can achieve.
The cURL command is used to download or upload data to the server using one of its more than 20 supported protocols. This data can be a file, email, or web page.
table of Contents
-
What is the cURL command?
-
download file
-
Follow redirect
-
Stop and continue download
-
Specify timeout
-
Use username and password
-
Use proxy
-
Download large files in chunks
-
Client certificate
-
Silent cURL
-
Get title
-
Multiple headers
-
Publish (upload) files
-
send email
-
Read email
-
Difference between cURL and wget
What is the cURL command?
cURL is an ideal tool for interacting with websites or APIs, sending requests to terminals and displaying responses or recording data to files. Sometimes, it is used as part of a larger script to pass the retrieved data to other functions for processing.
Since cURL can be used to retrieve files from the server, it is usually used to download a part of a website. It can perform this function well, but sometimes the wget command is more suitable for this job.
Later in this article, we will introduce some similarities and differences between wget and cURL.
We will show you how to start using cURL in the following section.
download file
The most basic command we can give to cURL is to download a website or file. Unless we specify a different protocol, cURL will use HTTP as its default protocol. To download the website, simply issue the following command:
$ curl http://www.google.comOf course, please enter any website or page to be retrieved.
Executing basic commands like this without other options is rarely useful, because it only tells cURL to retrieve the source code of the page you provided.
When we run commands, our terminal is full of HTML and other web script code-in this form, it is not particularly useful to us.
Let's download the website as an HTML document so that the content can be displayed, and the -output option can be added to implement cURL.
$ curl www.likegeeks.com --output likegeeks.htmlNow, the website we downloaded can be opened and displayed in the browser.
If you want to download an online file, the commands are roughly the same. But make sure to append the -output option to cURL, as we did in the example above.
If you do not do this, cURL will send the binary output of the online file to your terminal, which may cause it to malfunction.
This is what it looks like when we start downloading a 500KB Word document.
The Word document starts to download, and the current progress of the download is displayed in the terminal. After the download is complete, the file will be available in the directory where we saved the file.
In this example, the directory is not specified, so save it to the current working directory (the directory where the cURL command is run).
Also, did you notice the -L option we specified in the cURL command? In order to download this file, it is necessary, we will introduce its function in the next section.
Follow redirect
If the output is empty when trying to generate a URL for a website, it may mean that the website tells cURL to redirect to another URL. By default, cURL will not follow the redirect, but you can use the -L switch to tell it.
$ curl -L www.likegeeks.comIn the research in this article, we found that it is necessary to specify -L on most websites, so be sure to remember this little trick. By default, you may even want to attach it to most cURL commands.
Stop and continue download
If your download is interrupted, or you need to download a large file but do n’t want to complete all the operations at once, cURL provides an option to continue the transmission from where it was interrupted.
To stop the transmission manually, you can use the ctrl + c key combination to end the cURL process, just like stopping almost all currently running processes.
Our download has started, but was interrupted by ctrl + c. Now let's resume the download using the following syntax:
$ curl -C - example.com/some-file.zip --output MyFile.zip-C switch is the device that restores our file transfer, but also note that it is followed by a dash (-). This tells cURL to continue the file transfer, but to achieve this step, we must first check the part that has been downloaded, and find the last byte of the download to determine where to recover from.
At this time our file transfer has been resumed and we can continue to complete the download.
Specify timeout
If you want cURL to continue working for a while after you do n’t do an operation, you can specify a timeout in the command. This is especially useful because some operations in cURL do not have a timeout by default, so if you do n’t want If it hangs indefinitely, you need to specify one.
You can use -m switch to specify the maximum time required to execute the command. After the specified time, cURL will quit the operation being performed even if it is downloading or uploading a file.
cURL will specify the maximum time you want in seconds. Therefore, to timeout after one minute, the command will look like this:
$ curl -m 60 example.comAnother type of timeout you can specify with cURL is the amount of time spent on the link. This helps ensure that cURL does not spend too much time trying to connect to a host that is offline or inaccessible.
It also accepts seconds as a parameter. This option is written as –connect-timeout.
$ curl --connect-timeout 60 example.com
Use username and password
You can use -u switch to specify the username and password in the cURL command. For example, if you want to authenticate through an FTP server, the syntax is as follows:
$ curl -u username:password ftp://example.comYou can use it for any protocol, but FTP is usually used for such simple file transfers.
If we want to download the file shown in the screenshot above, we just need to issue the same command using the full path of the file.
$ curl -u username:password ftp://example.com/readme.txt
Use proxy
Before connecting to the host, it is easy to use cURL directly to use the proxy. cURL will use the HTTP proxy by default unless you specify otherwise.
Use -x switch to define the proxy. Since the protocol is not specified in this example, cURL will assume that it is an HTTP proxy.
$ curl -x 192.168.1.1:8080 http://example.comThis command will use 192.168.1.1 on port 8080 as a proxy to connect to example.com.
You can also use it with other agreements. The following is an example of using an HTTP proxy to transfer UR cURLL to an FTP server and retrieve files.
$ curl -x 192.168.1.1:8080 ftp://example.com/readme.txtcURL supports many other types of proxies and options for use with these proxies, but further expansion will be beyond the scope of this guide. Please refer to the cURL manual page for more information about proxy tunneling, SOCKS proxy, authentication, etc.
Download multiple files in chunks
We have shown how to stop and resume file transfer, but what if we want cURL to download only a part of the file? In this way, we can download multiple block files.
If you need to maintain a download limit or similar limit, you can download only certain parts of the file. The –range flag is used to complete this operation.
The size must be in bytes. Therefore, if we want to download the latest Ubuntu .iso file in a block size of 100 MB, the first command will look like this:
$ curl --range 0-99999999 http://releases.ubuntu.com/18.04/ubuntu-18.04.3-desktop-amd64.iso ubuntu-part1The second command will need to extract and download another 100 MB block in the next byte.
$ curl --range 0-99999999 http://releases.ubuntu.com/18.04/ubuntu-18.04.3-desktop-amd64.iso ubuntu-part1
$ curl --range 100000000-199999999 http://releases.ubuntu.com/18.04/ubuntu-18.04.3-desktop-amd64.iso ubuntu-part2Repeat this process until all blocks have been downloaded. The last step is to combine these blocks into a file, which can be done using the cat command.
$ cat ubuntu-part? > ubuntu-18.04.3-desktop-amd64.iso
Client certificate
To use certificate authentication instead of basic authentication to access the server, you can use the –cert option to specify the certificate file.
$ curl --cert path/to/cert.crt:password ftp://example.comcURL has many options for the format of the certificate file.
There are more options related to certificates: –cacert, –cert-status, –cert-type, etc. Please check the man page for a complete list of options.
Silent cURL
If you don't want to display cURL's progress meter and error message, -s switch can provide this function. It will still output the data you requested, so if you want the command to be 100% silent, you need to direct the output to a file.
Use this command with -O flag to save the file in the current working directory. This will ensure that cURL returns 0 output.
$ curl -s -O http://example.comAlternatively, you can use the –output option to choose a location to save the file and specify a name.
$ curl -s http://example.com --output index.html
Get title
Using cURL to get the title of the remote address is very simple, you only need to use the -I option.
$ curl -I example.comIf you use this option in conjunction with the –L option, cURL will return the headers for each address to which it redirects.
$ curl -I -L example.com
Multiple titles
You can use the -H option to pass the header to cURL. To pass multiple headers, you only need to use the -H option multiple times. This is an example:
$ curl -H 'Connection: keep-alive' -H 'Accept-Charset: utf-8 ' http://example.com
Publish (upload) files
POST is a common way for websites to accept data. For example, when you fill out a form online, it is likely to use the POST method to send data from the browser. To send data to the website in this way, use the -d option.
$ curl -d 'name=geek&location=usa' http://example.comTo upload a file instead of text, the syntax should be as follows:
$ curl -d @filename http://example.comUse multiple -d flags as needed to specify all different data or file names to be uploaded.
If you want to upload the file to an FTP server, you can use the -T option.
$ curl -T myfile.txt ftp://example.com/some/directory/
send email
Sending an email simply uploads the data from your computer (or other device) to the email server. Since cURL can upload data, we can use it to send emails. There are many options, but here is an example of how to send an email through an SMTP server:
$ curl smtp://mail.example.com --mail-from [email protected] --mail-rcpt [email protected] –upload-file email.txtYour email file needs to be formatted correctly. like this:
$ cat email.txtFrom: Web Administrator <[email protected]>To: John Doe <[email protected]>Subject: An example emailDate: Sat, 7 Dec 2019 02:10:15John,Hope you have a great weekend.-Admin
As usual, more detailed and professional options can be found in the manual page of cURL.
Read email
cURL supports IMAP (and IMAPS) and POP3, both of which can be used to retrieve emails from the mail server.
Log in using IMAP like this:
$ curl -u username:password imap://mail.example.comThis command will list available mailboxes, but will not view any specific messages. To do this, use the –X option to specify the UID of the message.
$ curl -u username:password imap://mail.example.com -X 'UID FETCH 1234'
Difference between cURL and wget
Sometimes people confuse cURL and wget because they can both retrieve data from the server. But this is the only thing they have in common.
In this article, we show the function of cURL. wget provides a different set of functions. wget is the best tool for downloading websites, and can recursively traverse directories and links to download entire websites.
To download the website, use wget. If you use a protocol other than HTTP or HTTPS or use it to upload files, please use cURL. cURL is also a good choice for downloading individual files from the web, but wget also does a great job.
