First, knowledge ready
- There is often required parameters set in nginx optimization, tcp_nodelay
- The core function parameters, the packet is composed of larger packages, improve bandwidth utilization is famous nagle algorithm
- tcp protocol, there is a phenomenon: Application layer data may be low (for example 1 byte), the transport layer overhead and 40 bytes (20 bytes IP header + 20 bytes of TCP header). In this case most of the control packet transmission, both increased bandwidth consumption, bandwidth utilization is not high
- nagle algorithm is to solve this problem. Before the data sent out has not been confirmed, or have not received ack before the end of the newly generated packet it is not allowed to be sent, it is necessary to scrape together a full MSS or send wait until confirmation is received until the time-out
Second, prepare the environment
| Package | version |
|---|---|
| THE | 18.04.1 LTS |
| docker | 18.06.0 it |
Client: 192.168.17.171
server: 192.168.17.173
Third, open the nagle algorithm
192.168.17.173, first prepare a nginx configuration file, and open nagle algorithm, set tcp_nodelay off;
root@k8s-node2:/tmp# more nginx.conf user nginx; worker_processes 1; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; tcp_nodelay off; keepalive_timeout 65; include /etc/nginx/conf.d/*.conf; }
Start container
root@k8s-node2:/tmp# docker run -d --name nginx_delay -v /tmp/nginx.conf:/etc/nginx/nginx.conf -p 80:80 nginx:latest 6b7d5a5d3c3ed021fed6847d138837754c5732979d1c829ec62107ec80440db8 root@k8s-node2:/tmp# docker ps | grep nginx_delay 6b7d5a5d3c3e nginx:latest "nginx -g 'daemon of…" 7 seconds ago Up 6 seconds 0.0.0.0:80->80/tcp nginx_delay
First, the use of the machine 80 fetch flow port tcpdump:
root@k8s-node2:/tmp# tcpdump -i ens3 port 80 -afexnnvv -w nginx_ab.cap
In 192.168.17.171, using the metrology tool ab pressure port volume
Note: You must use the -k parameter, use keepalived mode to simulate the nagle algorithm
root@k8s-node2:~# ab -n 1000 -c 100 -k http://127.0.0.1/ This is ApacheBench, Version 2.3 <$Revision: 1807734 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ ... Time per request: 44.619 [ms] (mean) ...
Filter out a lot of information, we came to this indicator Time per request, no matter how the test, with an average delay of about 40ms always
We look at the capture of information, the use of open wireshark
In a large number of data packets, we first look at the data processing packet, a SYN random selection, select the corresponding flow syn tcp
Select a fragment analysis
● In Linux, by default the delay confirmation, the so-called delayed acknowledgment is not received every request sent once ack, but wait a period of time, if this time happen to have packages to send, sit and "ride" together issued, or sent separately after a timeout. So the client will wait 40ms, then send the ACK
● Since nginx also set nagle algorithm, if not receive ack, it will wait for the arrival of the package, so it will appear like this
(1) 192.168.17.171 first sends an http get request (packet 677)
(2) transmits 192.167.17.173 PSH, ACK (packet 999)
(3) opening delay time due to confirm the default Linux, 192.168.17.171 waits 40ms, to see if the "ride"; and nginx on 192.168.17.173 due to the closure tcp_nodelay, it will be waiting for the arrival of re-ack response
after (4) 40ms, 192.168.17.171 not wait until the "ride", then send ack (1109 package)
(5) 192.168.17.173 transmitted http 200 (1118 packet) after receiving ACK
(. 6) 192.168.17.171 acknowledgment ACK after receiving a data transmission (packet number 1127)
192.168.17.171:47388 192.168.17.173:80
+-------+ +--------+
| | no.677 http get | |
| +---------------------------------> |
| | | |
| | no.999 PSH,ACK | |
| <---------------------------------+ |
| | | |
| | | |
| | | |
| | delay 40ms | |
| | | |
| | | |
| | | |
| | no.1109 ACK | |
| +---------------------------------> |
| | | |
| | no.1118 http 200 | |
| <---------------------------------+ |
| | | |
| | no.1127 ACK | |
| +---------------------------------> |
| | | |
| | | |
+-------+ +--------+Fourth, close nagle
Just set tcp_nodelay on;
root@k8s-node2:/tmp# sed -i '/tcp_nodelay/s/off/on/g' nginx.conf root@k8s-node2:/tmp# docker rm -f nginx_delay nginx_delay root@k8s-node2:/tmp# docker run -d --name nginx_delay -v /tmp/nginx.conf:/etc/nginx/nginx.conf -p 80:80 nginx:latest bac9bcf7a6e392a7a07afae165c3d5b4e3fb2fc43d3470f35802e12d1e7ae70d
Then ab test:
root@k8s-node2:~# ab -n 1000 -c 100 -k http://127.0.0.1/ This is ApacheBench, Version 2.3 <$Revision: 1807734 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ ... Time per request: 14.285 [ms] (mean) ...
Again capture the observations:
● Since the client is still open delay confirmation, so after 192.168.17.171 still not receive a timely response packet
● However nginx, tcp_nodelay on, it will immediately receive a response after 192.168.17.173 packet ack
after receipt ● 192.168.17.171 , have two data packet was not confirmed, and it will immediately send ack confirm:
(. 1) 192.168.17.171 first sends a http get request (packet 447)
in response psh immediately after (2) 192.168.17.173 received, ACK (packet No. 740)
(. 3) transmits 192.168.17.173 http 200 (package 741)
(. 4) 192.168.17.171 ACK response (packet 742)
192.168.17.171:49718 192.168.17.173:80
+-------+ +--------+
| | no.447 http get | |
| +---------------------------------> |
| | | |
| | no.740 PSH,ACK | |
| <---------------------------------+ |
| | | |
| | no.741 http 200 | |
| <---------------------------------+ |
| | | |
| | no.742 ACK | |
| +---------------------------------> |
| | | |
| | | |
+-------+ +--------+V. Summary
● This article reproduce the classic problem of 40ms
● mentioned in this article the two terms, nagle algorithm and delayed acknowledgments, they look very similar, but not the same. After acknowledgment for the delay is ack, ack will wait for "ride", if so, take the ride send, or wait for the timeout; nagle algorithm is required to wait until the end of the coming ack, or scrape together only after sending packets over a mss sent individually
● delayed acknowledgment in this article are the default Linux open feature, so in the experiment, the client will have a confirmed case of delay, the delay confirmation to close the client, you need to set setsockopt in TCP_QUICKACK
● this article focuses on is nginx the nagle algorithm, nagle algorithm is completely determined by the mechanism ack tcp agreement, if ACK reply soon to end, then, nagle in fact not much stitching data packets, although avoiding network congestion, the overall utilization of the network is still very low
● nagle algorithm in the case of mutual recognition of the role of the delay, the delay will have a serious effect, this is the need to guard against the
● nginx whether open nagle algorithm, depending on the business scene. For example, we see in the experiment:
(1) TCP_NODELAY OFF, will increase the communication delay, but will improve bandwidth utilization. High latency, large amount of data communication scenario should have good effect
(2) tcp_nodelay on, will increase the number of packets, but the response speed can be improved. The timeliness of high communication scenario should have good results