7 Concurrency Models for Single Point Servers

Learning video: In-depth understanding of the network I/O multiplexing concurrency model
in Linux

Socket Server Concurrency Model

Prerequisite knowledge:

• socket network programming
• Multiplexing IO multiplexing mechanism
• Concurrent programming theory such as multithreading/multiprocessing

Model 1: Single-threaded Accept (no IO multiplexing)

(1) Model structure diagram

(2) Model analysis

① The main thread main threadexecutes the blocking Accept, and each time the client Connect connects, main threadthe Accept responds and establishes the connection.

② The connection is successfully created, and after getting the Connfd1socket , the socket is still being main threadread and written serially, and the business is processed.

③ In ② processing business, if a new client Connectcomes over , there is Serverno response until all the business of the current socket is processed.

④ After the current client is processed, the connection is completed and the next client request is processed.

(3) Advantages and disadvantages

advantage:

• The socket programming process is clear and simple, suitable for learning to use and understand the basic socket programming process.

shortcoming:

• This model is not a concurrency model, but a serial server. At the same time, the maximum amount of network requests that listen and respond is 1, that is, the amount of concurrency is 1.
• Only suitable for learning basic socket programming, not suitable for any server Server construction.

Model 2: Single-threaded Accept + multi-threaded read and write services (no IO multiplexing)

(1) Model structure diagram

(2) Model analysis

① The main thread main threadexecutes the blocking Accept, and each time the client Connect connects, main threadthe Accept responds and establishes the connection.

② The connection is created successfully. After getting the Connfd1socket , a new thread is created thread1to process the client's read and write business. main threadStill back to Acceptblocking waiting for a new client.

③ thread1communicate Connfd1with the read and write.

④ In the process of business processing in ②, if a new client Connectcomes over , main threadthe server Acceptstill responds and establishes a connection, repeat ②.

(3) Advantages and disadvantages

advantage:

• Based on features that 模型一：单线程 Accept（无IO复用）support concurrency.
• Flexible use, one client corresponds to one thread for separate processing, and the Serverprocessing business has a high degree of cohesion. No matter how the client writes, the server will have a thread to respond to resources.

shortcoming:

• As the number of clients increases, the number of threads to be opened also increases, and the number of client and server threads is 1:1related . Therefore, for high concurrency scenarios, the number of threads is bottlenecked by the hardware upper limit. Too many threads will also increase the switching cost of the CPU and reduce the utilization of the CPU.
• For a long connection , once the client has no business reading and writing, as long as it is not closed, the corresponding thread of the Server still needs to keep the connection (heartbeat, health monitoring and other mechanisms), which occupies connection resources and wastes thread overhead resources.
• It is only suitable for scenarios where the number of clients is small and the number is controllable.
• Only suitable for learning basic socket programming, not suitable for building as a server.

Model 3: Single-threaded multi-channel IO multiplexing

(1) Model structure diagram

(2) Model analysis

main thread① listenFdAfter the main thread is created, a multi-channel I/O multiplexing mechanism (such as select, epoll) is used to monitor IO status blocking. There is a Client1client Connectrequest, and the I/O multiplexing mechanism detects a ListenFdtrigger read event, Acceptestablishes a connection, and adds the newly connFd1generated to 监听I/O集合it.

② Client1Make a normal read and write business request again, and main threadthe multiplexing I/O multiplexing mechanism will block and return, which will trigger the read/write event of the socket, etc.

③ For Client1the read and write business of , the Server is still in the main threadexecution process and continues to execute. If there is a new client Connectconnection , the Server will not respond immediately.

④ Wait until the server has finished processing a connection Read + Writeoperation , continue to return to the multi-channel I/O multiplexing mechanism to block, and repeat the process of ② and ③ for other connections.

(3) Advantages and disadvantages

advantage:

• The single process solves the model that can monitor the read and write status of multiple clients at the same time, and does not need to be 1:1related to the number of threads on the client.
• Multiple I/O multiplexing blocking, non-busy query state, no waste of CPU resources, high CPU utilization.

shortcoming:

• Although it can monitor the read and write status of multiple clients, at the same time, only one client's read and write operations can be processed. In fact, the read and write services are concurrent 1.
• When multiple clients access the Server, the business is executed in series, and a large number of requests will have queue delays. As shown in Figure ⑤, when Client3the main threadprocess is occupied, the Client1, Client2process is stuck IO复用waiting for next monitoring trigger event.

Model 4: Single-threaded multi-channel IO multiplexing + multi-threaded business work pool

(1) Model structure diagram

(2) Model analysis

main thread① listenFdAfter the main thread is created, a multi-channel I/O multiplexing mechanism (such as select, epoll) is used to monitor IO status blocking. There is a Client1client Connectrequest, and the I/O multiplexing mechanism detects a ListenFdtrigger read event, Acceptestablishes a connection, and adds the newly connFd1generated to 监听I/O集合it.

② When connFd1there is a readable message, the read event is triggered, and the message is read and written.

③ main threadRead the message according to a fixed protocol and hand it over to the Worker Poolworker thread pool. The worker thread pool has opened a fixed number of threads before the server starts thread. The threads in it only process message services and do not perform socket read and write operations.

④ The work pool finishes processing the business, triggers the connFd1write event, and writes the message of the receipt client to the main theadother party through .

Although this model improves the ability to process event services main thread, the efficiency is the same as that of model three because the entry and exit are both.

(3) Advantages and disadvantages

advantage:

• For 模型三, the business processing part is separated through the work pool to reduce multi-client access to the server. The business is executed serially, and a large number of requests will have a queuing delay time.
• In fact, the business concurrency of reading and writing is 1, but the business process concurrency is the number of Worker Pool threads, which speeds up the parallel efficiency of business processing.

shortcoming:

• Read and write are still handled main threadseparately , and the highest read and write parallel channel is still 1.
• Although multiple worker threads process services, they still need to be queued when they main threadare finally returned to the client, because the Read + Write.

Model 5: Single-threaded multi-channel IO multiplexing + multi-threaded multi-channel IO multiplexing (thread pool)

(1) Model structure diagram

(2) Model analysis

① Before the Server starts monitoring, it opens up a fixed number Nof threads and Thead Poolmanages it with a thread pool.

main thread② listenFdAfter the main thread is created, the multi-channel I/O multiplexing mechanism (such as select, epoll) is used to monitor the blocking of IO status. When there is a Client1client Connectrequest, the I/O multiplexing mechanism detects a ListenFdtrigger read event, Acceptestablishes a connection, and connFd1distributes to Thread Poola thread in for monitoring.

Thread PoolEach threadof ③ starts a multi-channel I/O multiplexing mechanism to monitor the successfully main threadestablished and distributed socket sockets.

④ As shown, thread1monitor ConnFd1、ConnFd2, thread2monitor ConnFd3, thread3monitor ConnFd4. When ConnFdthere is a read and write event corresponding to the socket, the corresponding thread handles the read, write and business of the socket.

(3) Advantages and disadvantages

advantage:

• main threadThe single-process reading and writing of is distributed to multiple threads to complete, which increases the number of parallel channels for reading and writing at the same time, and the number of parallel channels is the number Nof threadN pools .Thread
• The ConnFd 套接字number almost doubled. The previous total monitoring number depends on the maximum limit main threadof the multi-channel I/O multiplexing mechanism (select defaults to 1024, epoll defaults to memory size, ranging from about 3~6w), Therefore, the theoretical maximum number of concurrent responses for a single-point server is N * (3~6W)( the number ofN thread pools , and it is recommended to be proportional to CPU cores at 1:1).Thread
• If a good number of thread pools matches the number of CPU cores, you can try to bind CPU cores to Threads, thereby reducing the switching frequency of CPUs, improving the efficiency Threadof reasonable business, and reducing CPU switching costs.

shortcoming:

• Although the concurrent number of monitoring has increased, the highest read and write parallel channel is still N1, and multiple clients in the same Thread will experience read and write delays. In fact Thread, the 模型三：单线程多路IO复用consistent with .

Model 5 (process version) single-process multiplexing IO multiplexing + multi-process multiplexing IO multiplexing (process pool)

Nginx uses an improved version of this model.

(1) Model structure diagram

(2) Model analysis

There is 五、单线程IO复用+多线程IO复用（线程池）no .

Difference:

• The difference in the memory layout of processes and threads results in that main process(the main process) no longer Acceptoperates , but Acceptdisperses the process into 子进程(process)each .
• The characteristics of the process, the resources are independent, so main processif the Accept succeeds fd, other processes cannot share the resources, so each child process needs to accept and create a link by itself
• main processJust listen to the ListenFdstate , once the read event is triggered (there is a new connection request). Through some IPC (inter-process communication: such as signals, shared memory, pipes), etc., let the respective child processes Processcompete Acceptto complete the connection establishment and monitor each.

(3) Advantages and disadvantages

There is 五、单线程IO复用+多线程IO复用（连接线程池）no .

Difference:

• Multi-process memory resource space is slightly larger
• The multi-process model is more secure and stable, which is also due to the fact that the respective processes do not interfere with each other.

Model 6: Single-threaded multi-channel IO multiplexing + multi-threaded multi-channel IO multiplexing + multi-threading

(1) Model structure diagram

(2) Model analysis

① Before the Server starts monitoring, it opens up a fixed number Nof threads and Thead Poolmanages it with a thread pool.

main thread② listenFdAfter the main thread is created, the multi-channel I/O multiplexing mechanism (such as select, epoll) is used to monitor the blocking of IO status. When there is a Client1client Connectrequest, the I/O multiplexing mechanism detects a ListenFdtrigger read event, Acceptestablishes a connection, and connFd1distributes to Thread Poola thread in for monitoring.

Thread PoolEach threadof ③ starts a multi-channel I/O multiplexing mechanism to monitor the socket sockets that are successfully main threadestablished and distributed. Once one of the monitored client sockets is triggered I/O读写事件, a new thread will be opened immediately to process the I/O读写business .

④ When a reading and writing thread completes the current reading and writing business, if the current socket is not closed, then the current client socket (such as: ConnFd3) is added back to the monitoring thread of the thread pool, and its own thread is self-destructed.

(3) Advantages and disadvantages

Advantages :

• On the 模型五、单线程IO复用+多线程IO复用(连接线程池)basis , in addition to ensuring simultaneous response 最高并发数, it can also solve the problem of 读写并行通道limitations .
• Read and write parallel channels at the same time, 最大化极限so that one client can handle the read and write business corresponding to a single execution process, and the 1:1relationship between .

Disadvantages :

• This model is too ideal because it requires a large enough number of CPU cores.
• If the number of hardware CPUs is countable (the current hardware situation), then this model will cause a lot of wasted CPU switching cost. Because in order to ensure 1:1the , the server needs to open up the same Threadnumber as the client, so the number of CPUs bound to the listening thread pool for multiple I/O multiplexing in the thread pool will become zero. significance.
• This model would be optimal if each temporary read and write Threadcould be bound to a separate CPU. However, the current number of CPUs cannot reach an order of magnitude with the number of clients, and it is not even a few orders of magnitude worse.

Summarize

The above seven server processing models each have their own characteristics and advantages. For models dealing with high concurrency and high CPU utilization, most models are currently model five (or model five process version, such as Nginx is an improved version of model five process version).

As for the concurrency model, the design is not as complicated as possible, nor is it that the more threads are developed, the better. At the same time, it is necessary to consider the overhead of hardware utilization and switching costs. Model 6 is extremely complex in design and has many threads, but today's hardware capabilities cannot support it, which leads to extremely poor performance of the model. Therefore, it is necessary to choose a suitable model for different business scenarios, and it is not necessary to use a certain model.