In this lesson, we mainly understand the usage and common cases of Curl command. Before learning the content of this lesson, you need to understand the HTTP request process and the basics of the Linux operating system.
First of all, let's introduce Curl first. Curl is a tool in the Linux command line. It simulates client requests and follows the request protocol as HTTP or HTTPS. We learned about the developer tools under the Chrome browser in the previous course. It's a similar tool, except it's not an IDE tool, it's a command-line tool.
In fact, we need command-line tools in many scenarios. If you have a set of scripts and you need to call the HTTP API interface, or monitor the status of your own website server, you need to use commands to make HTTP\HTTPS request calls, and Get results.
In addition, when you need to use a tool to quickly analyze the failures or problems of a website, instead of using a browser, you also need a command line tool for easy use, so operation and maintenance engineers understand that the use of the Curl command is very necessary.
Use of Curl
The content of this class revolves around the Curl command, and shares the experience of using some scenarios.
Curl usage falls into two main categories.
One is the analysis of website services. Engineers will want to know the return status of the website; hope to understand the entire HTTP communication process; want to know specific scenarios such as IPV6 service site environment for simulation testing, and so on.
The second type is the use of functional purposes. Here we will learn how Curl uploads and downloads files; how to resume uploads, how to initiate multiple HTTP requests; how to use Curl's proxy mode for access.
Website service analysis
Let's start with the service analysis of the website. When talking about the status analysis of the website, some students often ask me some questions, such as how to troubleshoot if my website cannot be opened? I installed a service by myself, but I can’t request it when I open the browser, how can I analyze it? Asking such a question, I think you may not have a very clear idea of troubleshooting the entire website service, let's sort it out below.
After deploying and changing a site, the server status should be analyzed first, starting with the process:
- Let's see if the process is started? Does it exist?
- Is the status of the process normal? Is it running or has it entered a deadlock state?
In addition, we can use it to analyze whether the process is within the normal range through its utilization of CPU and other resources, and analyze its error information through logs.
After analyzing that the process status is normal, we need to understand why there is a problem accessing the server from the client?
- First, check from the underlying network, whether your network status can be pinged, and second, is it normal to establish a TCP connection? At this time, we can use
Telnet or TCPdump to capture or analyze packets. - After identifying the normal status of the network, look for problems at the upper layer, that is, the application layer. If the website serves HTTP\HTTPS web services, then we need to simulate HTTP
requests to the server to determine whether the header information and body information returned by the server meet the requirements. We expect that at this time, we need to use relevant tools for analysis. The Curl
command plays a very good role in this process.
Speaking of checking the HTTP\ HTTPS service status of the website, what aspects does the Curl command mainly check for this analysis?
The most common scenario is to look at the status returned by the entire server. The head header information contains HTTP status information (Status Code). If the value is 200, it means a normal response. If it is an abnormal response, you will see other values, such as 500, 404, 403, etc. We can use this status code to directly Locate a direction that may cause problems, such as: 403 may be due to restricted access, 404 may be a file not found, 500 may be a problem with the program, etc.
Sometimes, we choose to analyze whether the content of the body it returns meets expectations. This can all be achieved with the Curl command.
Curl command usage format
Next, let's take a look at how to use Curl.
First, add an options after the Curl command, which means configuring the parameters for Curl to initiate the request (optional), and finally add the URL that needs to be requested. Here I add a -i option, which means that the terminal only looks at the header information returned by the server without adding -i Then view the returned body data content by default. If a -o output.txt is added, it means to output the content returned by the server to a file.
Let's show how.
First, I use the Curl command to request baidu.com. At this time, we will see that it returns only the HTML code in the body. What if we only want to see the response head information it returns? We can add a -I option, and we will see the relevant header information returned by the server, which contains a lot of response head information. For example, the HTTP protocol type is 1.1, and the server status code is 200, indicating that the server returns normally. At the same time, you can see the data length, type and cached header information returned by the server.
The above is the common mode used by Curl, presumably most students should be more clear. Next, let's take a closer look at some special scenarios.
Here I request my blog address URL (command: curl www.taida.ltd), and you will find that there is no default body data returned. why? Because the server returns some redirection status codes, such as 301 and 302 redirection, there is no body data, only the header information is returned, so we can look at the content of the header information first.
At this time, if you want to understand the entire communication process more comprehensively, you can add a -v (command: curl www.taida.ltd) parameter. Can you see if the request server has sent it? And you can see what the content of the request sent by the client is, and what is returned by the server. The function of adding the -v parameter is to print out the entire communication process.
Here you can see the entire request process after adding -v, including: what is the header information of the request, and what is the header information returned by the server, and if there is body information returned by the server, the body information will also be printed out.
In the header information, we may need to focus on the status code of the service status. Usually you can analyze according to the following rules:
If the status code is between 100 and 199, it means that the server has normally given an information response, but has not fully established a complete HTTP request response.
200~299 means a successful response. Seeing this status code, it can be considered that the server is processing this request normally.
300~399 means redirection, such as 301, 302, these common redirection status codes.
If it is more than 400, it means that the client’s request method cannot get a normal response, then we need to analyze its status code in detail, such as 403, 404, 499, 400 and other related error codes. If you don't understand, you can search for the error code on Baidu or Google to see what it means.
500+ is generally a problem with the server program or some logic-related service responses.
Next, let's go back to the console for further demonstrations.
Through the -v parameter (curl -v www.taida.ltd), you can see that the status code returned by the server is 301, which means that a redirected status code is returned, and the header information of the location will be given. We know that the status code of 301 indicates that the request is directly redirected to another address, and the address after location is the redirected address (Location: http://www.taida.ltd/jeson). At this time, is it possible to request this address in one step in the Curl command? At this time, it can be realized through the -L parameter.
Here I add a -L parameter (curl -L http://www.taida.ltd) to see the effect. At this time, we will see that the entire communication process has changed, and then a lot of body information is printed out. Let's go back to the top of the printed information and have a look at the communication process.
First of all, what you see is that the first time the server returns a redirection status (301 redirection), Curl initiates the request again to the redirected address, and finally gets the body data and the response code of the last request.
Next, let's continue to explain other Curl scenarios in depth on the console, and how to add request header information with Curl requests.
First of all, you can see here is a link to download the data package, because it is a download package (large amount of data), so I add a -I parameter, only look at the returned head header information, the status code returned by the server after the request is initiated is 404 , indicating that the package was not found. So why is it returning a 404 here? Because I made a special security setting for this request link, I directly set a return 404 rule in Nginx, requiring the client to bring the Referer information before it can access and download normally. We know that Referer is a common information header in the HTTP request header, which indicates which address the request comes from.
So next, I add a Referer header request to the Curl request, here I add a -e option (curl -e "http://www.taida.ltd/" -I http://www.taida.ltd/download /ywgs_lg.tgz
), and then add a required refer link address URL, so that it matches the rules of the server and can be accessed normally. .
Let's take a look, what is returned at this time is a 200 status code.
How to use the Curl command
Exit the console demo, let's take a look at more ways to use the Curl command.
If you are visiting an HTTPS site, HTTPS sites are divided into public certificates and custom certificate types. If it is a public certificate, you do not need to add a private key locally. If it is a custom certificate created by you, you need to put the key on the client at this time, and then request the HTTPS server by adding -E to Curl.
For other uses, the Referer header was added in the demonstration just now, and you can also add other HTTP protocol header information. The HTTP request header can be customized or rewritten by the user. When requesting a certain website here, I added a --user-agent header, which indicates what type of browser the browser side of my request is using.
We saw that the header information in the previous explanation is the standard header information of HTTP. Can I add some custom request header information? If so how? At this time, we can add custom header information through --head, here is a demonstration of adding a Content-Type: application/json is the type header information of the request.
The other part is to simulate the server site that is detecting IPv6.
We know that IPv6 is used to solve the shortage of IP addresses for IPv4. At present, many large websites are already mandating the use of IPv6 and supporting the IPv6 protocol. We will also see that many base stations or their own IP addresses can be assigned an IPv6 address. If your local address has IPv6 addresses, and the server supports IPv6 services, it is very likely that your DNS resolution will prioritize IPv6 resolution. In this way, the conditions are met to detect IPv6 service sites, how to do it?
We can do it in such a way, as shown in the code, here is a specific explanation for you.
curl -6 -vo /dev/null --resolve "static.meituan.net:80:[240e:ff:e02c:1:21::]" "http://static.meituan.net/bs/@mtfe/knb-core/latest/dist/index.js"
curl -6 indicates that an IPv6 request is initiated. -v indicates the actual communication process. -o means put the body data returned by the request into the local empty device, that is, the body data is not displayed, only the header information is displayed, which is equivalent to the function of -I. –resolve means to resolve the domain name and IP. I manually specified and resolved the request address of the domain name static.meituan.net to an IPv6 address, and then followed it with the requested URL.
This is the way of simulated detection of the entire IPv6. Continue to demonstrate. At this time, we can see the entire return status and request process.
Well, the above is part of Curl's investigation of the website.
Curl functional use
Next, I will take you to understand some functional usage methods other than using Curl for website analysis and troubleshooting scenarios.
First of all, the first method is the proxy mode. We see that the Curl command can support requests in the form of a proxy.
Suppose the IP of my proxy here is 200.200.200.200, then just add an -x, that is, request the server site through the machine of 200.200.200.200, which is a way to access through the proxy.
The other is the upload and download of files. We can upload files through such formats. You can also download files directly through the FTP protocol. If you add a -C, it will enable the breakpoint resume. You can see this picture about the breakpoint resume. The client and the server will get the index of this package when downloading and downloading. When there is network jitter or fluctuation If you download again, you can continue to download directly based on the value of this index and continue to download based on the range value of the original breakpoint length. Therefore, if you want to download a file stably, you can use the method of resuming uploads from breakpoints.
Curl's HTTP request method
The last part is to introduce the HTTP request method of the Curl command.
In the Curl we introduced above, the get method is used to obtain the content of the server. In addition to the get method, we also have HTTP request methods, including post, add, option and other related methods. But if we want to specify the request method, we can do it by adding the request method to the -X option.
For example, -X post means to POST a request to the server, arg1 is the name of the parameter, followed by the value of the parameter, and then the second parameter arg2 and the value of the parameter, in this case, the data information of the request is transmitted in the form of post . The other is the put method, we can also pass it through -X put. -d is the value I passed. If the delete method is used, it is to request the server and delete the identified resource of the URL.