Operating system diary-use the curl command to analyze the time-consuming situation of the request

Reposted from https://cizixs.com/2017/04/11/use-curl-to-analyze-request/

I encountered a problem in my work recently, and the response to a certain request was very slow, so I hope there is a way to analyze which step of the request takes longer, so as to further find the cause of the problem. After searching on the Internet, I found a very useful method. The curl command can help you analyze the time-consuming parts of the request.

The curl command provides the -w parameter, which is explained in the manpage as follows:

 -w, --write-out <format>
              Make curl display information on stdout after a completed transfer. The format is a string that may contain plain text mixed with any number of variables. The  format
              can  be  specified  as  a literal "string", or you can have curl read the format from a file with "@filename" and to tell curl to read the format from stdin you write
              "@-".

              The variables present in the output format will be substituted by the value or text that curl thinks fit, as described below. All variables are specified  as  %{vari‐
              able_name} and to output a normal % you just write them as %%. You can output a newline by using \n, a carriage return with \r and a tab space with \t.

It can print some information according to the specified format, which can use some specific variables, and supports \n, \t and \r escape characters. There are many variables provided, such as status_code, local_port, size_download, etc. In this article, we only focus on variables related to the request time (variables starting with time_).

First write the following content to the text file curl-format.txt:

➜  ~ cat curl-format.txt
    time_namelookup:  %{time_namelookup}\n
       time_connect:  %{time_connect}\n
    time_appconnect:  %{time_appconnect}\n
      time_redirect:  %{time_redirect}\n
   time_pretransfer:  %{time_pretransfer}\n
 time_starttransfer:  %{time_starttransfer}\n
                    ----------\n
         time_total:  %{time_total}\n

So what do these variables mean? Let me explain:

time_namelookup: When DNS domain name is resolved, it is the process of converting https://zhihu.com into an ip address.
time_connect: the time when the TCP connection is established, which is the time of the three-way handshake
time_appconnect: the time when the upper layer protocol such as SSL/SSH is connected, for example connect/handshake time
time_redirect: the time from the beginning to the last request transaction
time_pretransfer: the time from the start of the request to the start of the response
time_starttransfer: the time from the start of the request to the first byte to be transferred
time_total: all the time spent in this request Time
Let's take a look at a simple request, no redirection, and no SSL protocol time:

➜  ~ curl -w "@curl-format.txt" -o /dev/null -s -L "http://cizixs.com"
    time_namelookup:  0.012
       time_connect:  0.227
    time_appconnect:  0.000
      time_redirect:  0.000
   time_pretransfer:  0.227
 time_starttransfer:  0.443
                    ----------
         time_total:  0.867

You can see that the time of each step of this request is printed out, and the unit of each number is seconds (seconds), so that you can analyze which step is more time-consuming and facilitate locating the problem. The meaning of each parameter of this command:

-w: read the format of the information to be printed from the file
-o /dev/null: discard the content of the response, because we don't care about it here, only the time-consuming situation of the request
-s: do not print the progress bar
from this Output, we can calculate the time of each step:

DNS query: 12ms
the TCP connection time: pretransfter (227) - namelookup ( 12) = 215ms
server processing time: starttransfter (443) - pretransfer ( 227) = 216ms
content transmission time: total (867) - starttransfer ( 443) = 424ms
to A more complicated one, visiting a certain homepage, with redirection and SSL protocol in the middle:

➜  ~ curl -w "@curl-format.txt" -o /dev/null -s -L "https://baidu.com"
    time_namelookup:  0.012
       time_connect:  0.018
    time_appconnect:  0.328
      time_redirect:  0.356
   time_pretransfer:  0.018
 time_starttransfer:  0.027
                    ----------
         time_total:  0.384

You can see that both time_appconnect and time_redirect are not 0, and the SSL protocol processing time is 328-18=310ms. And the time of pretransfer and starttransfer is shortened, this is the time of the request after redirection.