This article directory
Through the detailed graphic analysis of the iPerf3 -P parameter , we know that in the following two cases, we may use the -P parameter to optimize the test parameters to achieve the most accurate network test results.
- Test: Test scenario of "multiple connections between client and server"
- The TCP receive window and send window are too small to fully utilize the network bandwidth
The first scenario is the standard scenario where we use the -P parameter, and in the second scenario, we usually need to adjust the size of the TCP receiving window and sending window through the -W parameter to make full use of the network bandwidth. This article describes how iperf3 manages and processes the sending and receiving packet testing process of multiple connections in the scenario of "iperf3 -P multi-connection flow parallel test" on the client and receiving end.
From the following articles, we have analyzed the main process of state machine transition and code invocation.
- Seven, iperf3 source code analysis: state machine and state transition process —> server code when running forward TCP one-way test
- Eight, iperf3 source code analysis: state machine and state transition process —> client code when running forward TCP one-way test
- Nine, iperf3 source code analysis: detailed explanation of the interaction process between the server and the client during the forward TCP one-way test
1. Overall code structure and business process
In general:
1.1. For the server:
-
Creation process:
After entering the CREATE_STREAM state, the iperf3 server will first call iperf_accept()—>iperf_tcp_accept() to establish a test connection test stream after receiving the test TCP connection establishment request sent by the client, and then call iperf_new_stream( )—>iperf_init_stream()—>iperf_add_stream() creates and adds the current test connection test stream to the test instance. Note that this test connection test stream depends on the setting of the -P parameter, and there can be one or more . -
Management and use process:
After entering the TEST_RUNNING state, the iperf3 server will use select to poll all test connection test streams , and call iperf_recv()—>iperf_tcp_recv() on all sockets of est streams that have received test data. To collect test data and make test result statistics. Until it receives the TEST_END command sent by the client, it enters the TEST_END state. -
Release process:
After entering the IPERF_DONE state, the iperf3 server will call iperf_reset_test()—>iperf_free_stream() to traverse all test connection test streams, and release the test connection test streams one by one until all test streams are released.
1.2. For the client:
-
Creation process:
After receiving the CREATE_STREAM command sent by the server, it enters the CREATE_STREAM state, and the iperf3 client will actively initiate a TCP connection establishment request, and call iperf_new_stream—iperf_init_stream—iperf_add_stream until all test connection test streams are established successfully, and hang up Go to the test instance, and note that the test link test stream depends on the setting of the -P parameter, and there can be one or more . -
Management and use process:
After receiving the TEST_RUNNING command sent by the server and entering the TEST_RUNNING state, the iperf3 client will poll all test connection test streams and call iperf_send—iperf_tcp_send to send test data on each stream , and make test result statistics. until the test is completed. -
Release process:
After entering the IPERF_DONE state, the iperf3 client will call iperf_free_test—iperf_free_stream to traverse all test connection test streams, and release the test connection test streams one by one until all test streams are released.
2. Data structure analysis:
The test steam is a pointer under the test structure. This pointer points to a one-way list and manages one or more test streams under the current test instance. Each test stream corresponds to a TCP test connection.
struct iperf_test
{
char role; /* 'c' lient or 's' erver */
enum iperf_mode mode;
...... //此处省略很多代码
SLIST_HEAD(slisthead, iperf_stream) streams;//这里通过SLIST_HEAD这个宏定义了指向test->streams的单列表指针
SLIST_HEAD(plisthead, protocol) protocols;
...... //此处省略很多代码
};
At this time, struct slisthead {struct iperf_stream *slh_first;} streams after the macro expansion of SLIST_HEAD defines a variable streams of type struct slisthead, and this variable has only one member, which is the pointer slh_first of type struct iperf_stream.
It can be accessed through streams.slh_first .
3. Several important management functions and period function analysis:
3.1. For the server:
-
Creation process:
iperf_run_server()—>iperf_tcp_accept()
iperf_run_server()—>iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()
accepts the TCP connection initiated by the client, initiates and saves each TCP connection instance. -
Management and use process:
iperf_run_server()—>iperf_recv()—>iperf_tcp_recv(), query each TCP connection, and call iperf_tcp_recv() to receive all data on the connection if there is data received: -
Release process:
run()—>iperf_reset_test()—>iperf_free_stream(), reset the test object, release all test connections, and wait for the next test to start
3.2. For the client:
-
Creation process:
iperf_create_streams()—>create_socket()—>iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()
After calling iperf_create_streams, loop create_socket to initiate a TCPI test connection to the server, and create a TCP test connection instance, each The TCP test connection corresponds to an instance, and is stored in test->streams.slh_first in a single-table manner. -
Management and use process:
iperf_run_client()—>iperf_send()—>iperf_tcp_send(), cyclically send the data in each TCP test connection until the end of this test. -
Release process:
main()—>iperf_free_test()—>iperf_free_stream(). After completing this test, return to the main function and start releasing.
4. The macro used to manage the list:
The macros for list management are all placed in the queue.h file, which mainly performs operations such as initialization, addition, deletion, release, search, and traversal on the unit list of test->streams.slh_first. The following excerpts are part of the code and made some comments, which can be used as a reference for reading the code.
/*
* Singly-linked List definitions.
*/
// 定义一个单列表变量指针
#define SLIST_HEAD(name, type) \
struct name {
\
struct type *slh_first; /* first element */ \
}
#define SLIST_HEAD_INITIALIZER(head) \
{
NULL }
#define SLIST_ENTRY(type) \
struct {
\
struct type *sle_next; /* next element */ \
}
/*
* Singly-linked List access methods.
*/
//查找单列表头结点
#define SLIST_FIRST(head) ((head)->slh_first)
#define SLIST_END(head) NULL
#define SLIST_EMPTY(head) (SLIST_FIRST(head) == SLIST_END(head))
#define SLIST_NEXT(elm, field) ((elm)->field.sle_next)
//遍历单列表
#define SLIST_FOREACH(var, head, field) \
for((var) = SLIST_FIRST(head); \
(var) != SLIST_END(head); \
(var) = SLIST_NEXT(var, field))
#define SLIST_FOREACH_PREVPTR(var, varp, head, field) \
for ((varp) = &SLIST_FIRST((head)); \
((var) = *(varp)) != SLIST_END(head); \
(varp) = &SLIST_NEXT((var), field))
/*
* Singly-linked List functions.
*/
//初始化一个列表
#define SLIST_INIT(head) {
\
SLIST_FIRST(head) = SLIST_END(head); \
}
5. Analysis of running process
5.1. For the server:
- Creation process:
iperf_run_server()—>iperf_tcp_accept()
iperf_run_server()—>iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()
accepts the TCP connection initiated by the client, initiates and saves each TCP connection instance. Call Log as follows:
iperf3 -s
......
debug out: func = iperf_run_server ,line = 625, file = iperf_server_api.c
debug out: func = iperf_tcp_accept ,line = 123, file = iperf_tcp.c
debug out: func = iperf_run_server ,line = 727, file = iperf_server_api.c
debug out: func = iperf_new_stream ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream ,line = 4461, file = iperf_api.c
debug out: add stream 1
[ 5] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 56288
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server ,line = 625, file = iperf_server_api.c
debug out: func = iperf_tcp_accept ,line = 123, file = iperf_tcp.c
debug out: func = iperf_run_server ,line = 727, file = iperf_server_api.c
debug out: func = iperf_new_stream ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: add stream 3, i = 3
[ 8] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 56292
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server ,line = 625, file = iperf_server_api.c
debug out: func = iperf_tcp_accept ,line = 123, file = iperf_tcp.c
debug out: func = iperf_run_server ,line = 727, file = iperf_server_api.c
debug out: func = iperf_new_stream ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: search stream 3, i = 4
debug out: add stream 4, i = 4
[ 10] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 56304
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server ,line = 625, file = iperf_server_api.c
debug out: func = iperf_tcp_accept ,line = 123, file = iperf_tcp.c
debug out: func = iperf_run_server ,line = 727, file = iperf_server_api.c
debug out: func = iperf_new_stream ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: search stream 3, i = 4
debug out: search stream 4, i = 5
debug out: add stream 5, i = 5
[ 12] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 56314
---------------------------------------------------------------------------------------
- Management and use process:
iperf_run_server()—>iperf_recv()—>iperf_tcp_recv()
Call Log as shown below to query each TCP connection, and call iperf_tcp_recv() to receive all data on the connection if there is data received:
iperf3 -s
......
debug out: func = iperf_run_server ,line = 848, file = iperf_server_api.c
debug out: func = iperf_recv ,line = 1951, file = iperf_api.c
debug out: current sp->id =1
debug out: current sp->id =3
debug out: current sp->id =4
debug out: current sp->id =5
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server ,line = 848, file = iperf_server_api.c
debug out: func = iperf_recv ,line = 1951, file = iperf_api.c
debug out: current sp->id =1
debug out: receiving at sp->id =1
debug out: func = iperf_tcp_recv ,line = 59, file = iperf_tcp.c
debug out: current sp->id =3
debug out: current sp->id =4
debug out: current sp->id =5
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server ,line = 848, file = iperf_server_api.c
debug out: func = iperf_recv ,line = 1951, file = iperf_api.c
debug out: current sp->id =1
debug out: current sp->id =3
debug out: receiving at sp->id =3
debug out: func = iperf_tcp_recv ,line = 59, file = iperf_tcp.c
debug out: current sp->id =4
debug out: receiving at sp->id =4
debug out: func = iperf_tcp_recv ,line = 59, file = iperf_tcp.c
debug out: current sp->id =5
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server ,line = 848, file = iperf_server_api.c
debug out: func = iperf_recv ,line = 1951, file = iperf_api.c
debug out: current sp->id =1
debug out: receiving at sp->id =1
debug out: func = iperf_tcp_recv ,line = 59, file = iperf_tcp.c
debug out: current sp->id =3
debug out: receiving at sp->id =3
debug out: func = iperf_tcp_recv ,line = 59, file = iperf_tcp.c
debug out: current sp->id =4
debug out: current sp->id =5
debug out: receiving at sp->id =5
debug out: func = iperf_tcp_recv ,line = 59, file = iperf_tcp.c
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server ,line = 848, file = iperf_server_api.c
debug out: func = iperf_recv ,line = 1951, file = iperf_api.c
debug out: current sp->id =1
debug out: current sp->id =3
debug out: current sp->id =4
debug out: receiving at sp->id =4
debug out: func = iperf_tcp_recv ,line = 59, file = iperf_tcp.c
debug out: current sp->id =5
debug out: receiving at sp->id =5
debug out: func = iperf_tcp_recv ,line = 59, file = iperf_tcp.c
---------------------------------------------------------------------------------------
- Release process:
run()—>iperf_reset_test()—>iperf_free_stream()
debug out: func = run ,line = 179, file = main.c
debug out: func = iperf_reset_test ,line = 3072, file = iperf_api.c
debug out: func = iperf_free_stream ,line = 4222, file = iperf_api.c
debug out: free stream 1
debug out: func = iperf_free_stream ,line = 4222, file = iperf_api.c
debug out: free stream 3
debug out: func = iperf_free_stream ,line = 4222, file = iperf_api.c
debug out: free stream 4
debug out: func = iperf_free_stream ,line = 4222, file = iperf_api.c
debug out: free stream 5
5.2. For the client:
- Creation process:
iperf_create_streams()—>create_socket()—>iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()
After calling iperf_create_streams, loop create_socket to initiate a TCPI test connection to the server, and create a TCP test connection instance, each The TCP test connection corresponds to an instance, and is stored in test->streams.slh_first in a single-table manner.
debug out: func = iperf_create_streams ,line = 69, file = iperf_client_api.c
debug out: func = create_socket ,line = 129, file = net.c
debug out: func = iperf_new_stream ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream ,line = 4461, file = iperf_api.c
debug out: add stream 1
[ 5] local 127.0.0.1 port 56288 connected to 127.0.0.1 port 5201
debug out: func = create_socket ,line = 129, file = net.c
debug out: func = iperf_new_stream ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: add stream 3, i = 3
[ 7] local 127.0.0.1 port 56292 connected to 127.0.0.1 port 5201
debug out: func = create_socket ,line = 129, file = net.c
debug out: func = iperf_new_stream ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: search stream 3, i = 4
debug out: add stream 4, i = 4
[ 9] local 127.0.0.1 port 56304 connected to 127.0.0.1 port 5201
debug out: func = create_socket ,line = 129, file = net.c
debug out: func = iperf_new_stream ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: search stream 3, i = 4
debug out: search stream 4, i = 5
debug out: add stream 5, i = 5
[ 11] local 127.0.0.1 port 56314 connected to 127.0.0.1 port 5201
- Management and use process:
iperf_run_client()—>iperf_send()—>iperf_tcp_send()
loops to send the data in each TCP test connection until the end of this test.
debug out: func = iperf_run_client ,line = 633, file = iperf_client_api.c
debug out: func = iperf_send ,line = 1890, file = iperf_api.c
debug out: func = iperf_send ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send ,line = 87, file = iperf_tcp.c
debug out: sending at sp->id = 1
debug out: func = iperf_send ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send ,line = 87, file = iperf_tcp.c
debug out: sending at sp->id = 3
debug out: func = iperf_send ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send ,line = 87, file = iperf_tcp.c
debug out: sending at sp->id = 4
debug out: func = iperf_send ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send ,line = 87, file = iperf_tcp.c
debug out: sending at sp->id = 5
debug out: func = iperf_send ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send ,line = 87, file = iperf_tcp.c
debug out: sending at sp->id = 1
debug out: func = iperf_send ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send ,line = 87, file = iperf_tcp.c
debug out: sending at sp->id = 3
debug out: func = iperf_send ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send ,line = 87, file = iperf_tcp.c
debug out: sending at sp->id = 4
debug out: func = iperf_send ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send ,line = 87, file = iperf_tcp.c
debug out: sending at sp->id = 5
- Release process:
main()—>iperf_free_test()—>iperf_free_stream(). After completing this test, return to the main function and start releasing.
iperf Done.
debug out: func = main ,line = 121, file = main.c
debug out: func = iperf_free_test ,line = 2940, file = iperf_api.c
debug out: func = iperf_free_stream ,line = 4222, file = iperf_api.c
debug out: free stream 1
debug out: func = iperf_free_stream ,line = 4222, file = iperf_api.c
debug out: free stream 3
debug out: func = iperf_free_stream ,line = 4222, file = iperf_api.c
debug out: free stream 4
debug out: func = iperf_free_stream ,line = 4222, file = iperf_api.c
debug out: free stream 5