Use iperf3 to perform TCP test on long-fat network (TCP long-fat pipeline), how to configure iperf3

The current network usually has a large bandwidth. If there is a delay caused by some intermediate devices, it is easy to cause the problem of a long and fat network. Through the article "Long fat network and TCP long fat pipeline" , we already know that in the TCP long fat pipeline, the performance of TCP is very poor, and it is difficult to make full use of the network bandwidth. If we want to test such a network, we can get the maximum throughput of the network. Quantity, how should we use iperf3 correctly, how to correctly set the parameters of iperf3.

1. The two characteristics of the TCP long-fat pipeline

First of all, let's review the two major characteristics of TCP long-fat pipelines:

• The transmission delay (transmission delay) is very small
  . The speed of sending and receiving packets is very fast, and a large amount of data can be sent to the network in a very short time.
• The propagation delay is very large
  . It takes a long time (compared to the sending delay) for the data packet to be transmitted to the receiving end from the time it is sent to the network.

2. The problem of sending TCP connection data in the TCP long-fat pipeline

This will cause problems: before the first bit sent by the sender reaches the receiver, the sender has already sent the last bit, then the window becomes 0, and then stops sending data and waits for the receiver to bring it back through ACK After receiving the notification window, the window can be opened to continue sending, which makes TCP unable to occupy the full bandwidth to send data.

3. The method of using iperf3 to test the maximum throughput of the TCP connection in the TCP long-fat pipeline

Usually, when we stream in a network with a bandwidth of 1Gbps, we hope that the maximum TCP throughput can reach 1Gbps, but due to the above-mentioned long and fat network reasons, it may not reach 1Gbps.

There are two main ways to solve this problem:

3.1. If one TCP connection cannot occupy the entire bandwidth, then consider using multiple TCP connections to occupy the entire bandwidth.

• Use the -P parameter

  Assuming that the window is not enough due to long and fat pipes, and a TCP stream can reach a maximum of 100Mbps, the TCP receiving window is set separately for each TCP connection, that is, each TCP connection maintains its own receiving window. So we use the -P parameter to start 10 TCP connections at the same time. After stabilization, we can reach the maximum throughput (bandwidth) 10*100Mbps = 1Gbps.
  For details, see: iPerf3 -P parameter detailed graphic analysis

• The principle of using multiple iperf3 processes
  is the same as using -P, because each TCP connection maintains its own independent receiving window, so you can open more TCP connections.
  For details, see: Simultaneously start multiple iperf3 processes for large flow testing

3.2. Expand the TCP window to improve network utilization

• Use the -w parameter to expand the TCP window.
  For details, see iPerf3 -w parameter detailed graphic analysis

  Note that the actual value of the -w parameter is limited by different linux implementations. Some settings exceed the maximum value and return an error. When some settings exceed the maximum value, simply change the parameter to the maximum value and return success. Can be used under root authority, use the echo command to modify the maximum value of the system limit.

/proc/sys/net/core/wmem_max 
/proc/sys/net/core/wmem_default
/proc/sys/net/core/rmem_max 
/proc/sys/net/core/rmem_default

echo 425984 > /proc/sys/net/core/rmem_max