In a Linux server, various methods such as kernel tuning, DPDK, and XDP can be used to improve the server's anti-attack capability and reduce the impact of DDoS on normal services. In the application, you can use all levels of cache, WAF, CDN, etc. to mitigate the impact of DDoS on the application.
However, it should be noted that if the DDoS traffic has already reached the Linux server, even if various optimizations are made at the application layer, the network service delay will generally be much greater than usual.
Therefore, in practical applications, we usually use Linux servers with professional traffic cleaning and network firewall equipment to alleviate this problem.
In addition to the increase in network latency caused by DDoS, I think you must have seen network latency caused by many other reasons, such as:
-
Delay caused by slow network transmission.
-
Latency caused by slow packet processing in the Linux kernel stack.
-
Delays caused by slow application data processing, etc.
So what should we do when we encounter delays caused by these reasons? How can I locate the root cause of network latency? Let's discuss network latency in this article.
Linux network latency
When it comes to network latency (Network Latency), people usually think it refers to the time required for network data transmission. However, the "time" here refers to two-way traffic, that is, the round-trip time for data to be sent from the source to the destination, and then the response is returned from the destination address: RTT (Round-Trip Time).
In addition to network latency, another commonly used indicator is application latency (Application Latency), which refers to the time it takes for an application to receive a request and return a response. Typically, application latency, also known as round-trip latency, is the sum of network data transfer time plus data processing time.
Usually, people use the ping command to test the network delay. Ping is based on the ICMP protocol. It obtains the round-trip delay time by calculating the time difference between the response message sent by ICMP and the request message sent by ICMP. This process does not require special authentication, so it is often used by many network attacks, such as port scanning tool nmap, grouping tool hping3, etc.
Therefore, in order to avoid these problems, many network services disable ICMP, which makes it impossible to use ping to test the availability and round-trip delay of network services. In this case, you can use traceroute or hping3's TCP and UDP modes to get network latency.
For example:
# -c: 3 requests
# -S: Set TCP SYN
# -p: Set port to 80
$ hping3 -c 3 -S -p 80 google.com
HPING google.com (eth0 142.250.64.110): S set, 40 headers + 0 data bytes
len=46 ip=142.250.64.110 ttl=51 id=47908 sport=80 flags=SA seq=0 win=8192 rtt=9.3 ms
len=46 ip=142.250.64.110 ttl=51 id=6788 sport=80 flags=SA seq=1 win=8192 rtt=10.9 ms
len=46 ip=142.250.64.110 ttl=51 id=37699 sport=80 flags=SA seq=2 win=8192 rtt=11.9 ms
--- baidu.com hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 9.3/10.9/11.9 ms
Of course, you can also use traceroute:
$ traceroute --tcp -p 80 -n google.com
traceroute to google.com (142.250.190.110), 30 hops max, 60 byte packets
1 * * *
2 240.1.236.34 0.198 ms * *
3 * * 243.254.11.5 0.189 ms
4 * 240.1.236.17 0.216 ms 240.1.236.24 0.175 ms
5 241.0.12.76 0.181 ms 108.166.244.15 0.234 ms 241.0.12.76 0.219 ms
...
24 142.250.190.110 17.465 ms 108.170.244.1 18.532 ms 142.251.60.207 18.595 ms
traceroute sends three packets at each hop along the route, and prints the round-trip delay when a response is received. If there is no response or the response times out (default 5s), an asterisk * will be output.
Case presentation
We need to host two hosts host1 and host2 in this demo:
-
host1 (192.168.0.30): hosts two Nginx web applications (normal and delayed)
-
host2 (192.168.0.2): analysis host
host1 ready
On host1, let's run two containers, the official Nginx and the delayed version of Nginx:
# Official nginx
$ docker run --network=host --name=good -itd nginx
fb4ed7cb9177d10e270f8320a7fb64717eac3451114c9fab3c50e02be2e88ba2
# Latency version of nginx
$ docker run --name nginx --network=host -itd feisky/nginx:latency
b99bd136dcfd907747d9c803fdc0255e578bad6d66f4e9c32b826d75b6812724
Run the following command to verify that both containers are serving traffic:
$ curl http://127.0.0.1
<!DOCTYPE html>
<html>
...
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
$ curl http://127.0.0.1:8080
...
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
host2 ready
Now let's test their latency with hping3 mentioned above to see the difference. In host2, execute the following commands to test the latency of port 8080 and port 80 of the case machine respectively:
Port 80:
$ hping3 -c 3 -S -p 80 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=0 win=29200 rtt=7.8 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=1 win=29200 rtt=7.7 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=80 flags=SA seq=2 win=29200 rtt=7.6 ms
--- 192.168.0.30 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.6/7.7/7.8 ms
8080 port:
# 测试8080端口延迟
$ hping3 -c 3 -S -p 8080 192.168.0.30
HPING 192.168.0.30 (eth0 192.168.0.30): S set, 40 headers + 0 data bytes
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=0 win=29200 rtt=7.7 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=1 win=29200 rtt=7.6 ms
len=44 ip=192.168.0.30 ttl=64 DF id=0 sport=8080 flags=SA seq=2 win=29200 rtt=7.3 ms
--- 192.168.0.30 hping statistic ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.3/7.6/7.7 ms
From this output you can see that both ports have about the same latency at 7ms. But this only works for a single request. What if you switch to concurrent requests? Next, let's try it with wrk (https://github.com/wg/wrk).
Port 80:
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30/
Running 10s test @ http://192.168.0.30/
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.19ms 12.32ms 319.61ms 97.80%
Req/Sec 6.20k 426.80 8.25k 85.50%
Latency Distribution
50% 7.78ms
75% 8.22ms
90% 9.14ms
99% 50.53ms
123558 requests in 10.01s, 100.15MB read
Requests/sec: 12340.91
Transfer/sec: 10.00MB
8080 port:
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
Running 10s test @ http://192.168.0.30:8080/
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 43.60ms 6.41ms 56.58ms 97.06%
Req/Sec 1.15k 120.29 1.92k 88.50%
Latency Distribution
50% 44.02ms
75% 44.33ms
90% 47.62ms
99% 48.88ms
22853 requests in 10.01s, 18.55MB read
Requests/sec: 2283.31
Transfer/sec: 1.85MB
From the above two outputs, it can be seen that the average delay of the official Nginx (monitoring port 80) is 9.19ms, while the average delay of the case Nginx (monitoring port 8080) is 43.6ms. From the perspective of delay distribution, the official Nginx can complete 90% of the requests within 9ms; for the case of Nginx, 50% of the requests have reached 44ms.
So what's going on here? Let's do some analysis:
In host1, let's use tcpdump to capture some network packets:
$ tcpdump -nn tcp port 8080 -w nginx.pcap
Now, re-run the wrk command on host2
$ wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
When the wrk command is complete, switch back to Terminal 1 (host1's terminal) again and press Ctrl+C to end the tcpdump command. Then, use Wireshark to copy the captured nginx.pcap to the local machine (if VM1 (host1's virtual machine) already has a graphical interface, you can skip the copying step), and use Wireshark to open it.
Due to the large number of network packets, we can filter them first. For example, after selecting a package, you can right-click and select "Follow" -> "TCP Stream", as shown in the figure below:
Then, close the dialog that pops up and return to the main Wireshark window. At this time, you will find that Wireshark has automatically set a filter expression tcp.stream eq 24 for you. As shown in the figure below (the source IP and destination IP are omitted in the figure):
From here, you can see every request and response for this TCP connection, starting with the three-way handshake. Of course, this may not be intuitive enough. You can continue to click Statistics -> Flow Graph in the menu bar, select "Limit to display filter", and set Flow type to "TCP Flows":
Note that the left side of this diagram is the client, while the right side is the Nginx server. It can be seen from this figure that the first three handshakes and the first HTTP request and response are quite fast, but the second HTTP request is relatively slow, especially after the client receives the first data packet from the server. The ACK response (blue line in the diagram) is sent after 40ms.
Seeing the value of 40ms, did you think of anything? In fact, this is the minimum timeout for TCP to delay an ACK. This is an optimization mechanism of TCP ACK, that is, instead of sending an ACK for each request, it waits for a period of time (such as 40ms) to see if there is a "ride" data packet. If there are other packets to send during this time, they will be sent along with the ACK. Of course, if you can't wait for other packets, the ACK will be sent separately after the timeout.
Since the client in the case has a 40ms delay, we have reason to suspect that the client has enabled the Delayed Acknowledgment Mechanism. The client here is actually the wrk that was running before.
According to the TCP documentation, the Fast Acknowledgment Mode (Fast Acknowledgment Mode) is only enabled when the TCP socket is specifically set with TCP_QUICKACK; otherwise, the delayed acknowledgment mechanism is used by default:
TCP_QUICKACK (since Linux 2.4.4)
Enable quickack mode if set or disable quickack mode if cleared. In quickack mode, acks are sent imme‐
diately, rather than delayed if needed in accordance to normal TCP operation. This flag is not perma‐
nent, it only enables a switch to or from quickack mode. Subsequent operation of the TCP protocol will
once again enter/leave quickack mode depending on internal protocol processing and factors such as
delayed ack timeouts occurring and data transfer. This option should not be used in code intended to be
portable.
Let's test our challenge:
$ strace -f wrk --latency -c 100 -t 2 --timeout 2 http://192.168.0.30:8080/
...
setsockopt(52, SOL_TCP, TCP_NODELAY, [1], 4) = 0
...
It can be seen that wrk only sets the TCP_NODELAY option, but does not set TCP_QUICKACK. Now you can see why there is a delay in delaying Nginx (case Nginx) responses.
in conclusion
In this article, I'll show you how to analyze increased network latency. Network latency is a core network performance metric. Due to the influence of various factors such as network transmission and network packet processing, network delay is inevitable. But too much network delay will directly affect user experience.
-
Use tools such as hping3 and wrk to confirm whether the network delay of a single request and concurrent requests is normal.
-
Using traceroute, confirm that the route is correct, and look at the latency of each gateway hop along the route.
-
Use tcpdump and Wireshark to confirm whether the network packets are sent and received normally.
-
Use strace, etc. to observe whether the application's call to the network socket is normal.
Link: https://zhuanlan.zhihu.com/p/525246901