cURL is everywhere. It is hidden in almost all devices such as cars, Blu-ray players, etc. It transfers any type of data over the Internet Protocol.
In this article, we'll demystify the cURL command-line tool, explain how it became a common code, and give examples of its usage.
What does cURL mean?
cURL (Client URL) is an open source command-line tool and a cross-platform library (libcurl) for transferring data between servers and distributed to almost all new operating systems. cURL programming is used almost anywhere data needs to be sent or received over the Internet Protocol.
cURL supports almost all Internet protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP).
History of cURL
Back in the dark ages of the 90s, when everyone was still using command-line tools, Daniel Sterberg wanted to develop a simple IRC script that would convert currency for chat room members. In 1997, there weren't many ways to build the foundations of Internet Protocol data transfer, so Httpget (a few hundred lines of code for HTTP-based transfers) became the origin of cURL. It was first known as HTTPGET 1.0 in honor of its foundation.
After a few months, support for FTP was developed, and the name had to be removed. Now, it's called urlget 2.0. After several updates, on March 30, 1998, the name was changed again to the now well-known cURL 3.0.
Before cURL, there was actually a similar wget. We won't go into too much detail, but the main difference between wget and cURL is their respective download capabilities, e.g. the former can resume from an interrupted transfer and continue the download.
What does cURL do?
cURL is designed to transfer data over the internet protocol. Everything else is out of scope. It doesn't even handle the transferred data, it just executes the transfer process.
cURL can be used for debugging. For example, using "curl -v https://oxylabs.io " can display detailed output of a connection request, including user agent, handshake data, port and other details.
There are too many cURL command options to list and explain. Fortunately, you can use the "curl --help" option, which lists all curl command line usages with short explanatory notes. Although there is no background knowledge about how to use cURL in these command line usage, users can also know some command line usage through the list.
How to use cURL?
Almost anyone with a relatively recent operating system can use cURL, since cURL comes as a default in Windows, MacOS, and most Linux distributions. For older systems, such as any Windows operating system prior to 10, it may be necessary to download and install cURL.
To use cURL, simply open a terminal and type "curl". Under normal circumstances, "curl -help" will automatically pop up, and the user can choose whether to execute the "curl –help" command line. As mentioned earlier, Help will list all command possibilities.
cURL commands can be combined by adding the listed command lines and typing the URL. Arguments can be short (eg -o, -L, etc.) or long (eg -verbose). The parameters are distinguished by using single or double dashes.
Curl accesses Taobao's product details API to obtain product details data
use cURL
send request
cURL is a powerful tool for data transfer over the Internet Protocol. Trying to detail what cURL can do and list all the options would be an impossible task. cURL can be used in a variety of ways. Here are some common common cases.
Since cURL was originally developed for HTTP, we can send all regular requests (POST, GET, PUT, etc.). In order to send a POST request to a URL, use the -d (or --data) command line. Most websites will reject such requests from unauthorized users, so we will use a fake API for testing.
curl -d “name=something&value=somethingelse”
https://jsonplaceholder.typicode.com/posts/
Sending such a request should return:
{
"name": "something",
"value": "somethingelse",
"id": 101
}
Here is a brief description of the whole process:
- curl starts our command
- -d is the "data" parameter of the POST request
- Quotes ("") begin our content declaration. Note that some operating systems will only accept single quotes, while others will accept double quotes.
- Finally there is the destination. URL syntax should always be exact as cURL does not automatically follow redirects.
We can also send POST requests in JSON format, but additional options must be provided to tell the server that we are sending JSON. cURL will not do any interpretation on behalf of the user and will send the default Content-Type header of application/text, so we have to add the header Content-Type: application/json ourselves.
curl -H "Content-Type: application/json" --data
"{\"data\":\"some data\"}"
https://jsonplaceholder.typicode.com/posts/
follow redirect
cURL does not automatically follow redirects. If we expect to do this, we should add an extra command line. Let's look at an example:
curl https://baidu.com
Our browsers handle redirects on their own, so we might not even notice a problem with such requests. However, if we send cURL to do the work, we get notified that the document has been moved while trying to connect. Let's take a look at the results returned by directly executing the above commands in the win10 command prompt:
We can see that the displayed result is 302 Found. Some experienced users should know that 302 Found is a status code (Status Code) in the HTTP protocol. It can be simply understood that the resource did exist originally, but it has been temporarily changed. location; or in other words, the resource is temporarily resolved to a certain URL. Usually a header is sent to temporarily redirect to a new location. This means that redirects are not automatically followed like regular browsers. So, in order for cURL to follow redirects, we have to add a special parameter "-L" (the parameter is case sensitive).
curl -L https://baidu.com
The result of the execution is as follows:
By now, we should have received the regular answer from Baidu, since cURL follows the redirection from https://baidu.com to https://www.baidu.com .
connect through proxy
cURL can be used to connect to any destination through a proxy. As with any other cURL statement, the URL, syntax and everything else remains the same, except for the added parameters and their properties.
curl --proxy proxyaddress:port
https://jsonplaceholder.typicode.com/
Entering a proxy and port after "-proxy" will connect via the entered address path. The proxy usually needs to submit detailed login credentials for authentication, and the user can complete the submission through the command line with the -U parameter.
curl --proxy proxy:port -U “username:password”
https://jsonplaceholder.typicode.com/
Certain websites will require authentication before accepting any connection requests. Server authentication uses a similar parameter: "-u".
curl -u username:password
https://jsonplaceholder.typicode.com/
We will introduce more about cURL in the next article.
in conclusion
cURL is a very powerful tool for Internet protocol transport. Mastering its usage is by no means easy, yet it can be an irreplaceable tool in any developer's toolkit. Frankly speaking, it is not difficult to expand on cURL use cases, but it is almost impossible to explore all possibilities, because there are too many possibilities involved.