Design of multi-threaded concurrent http tasks
Table of contents
Design of multi-threaded concurrent http tasks
1.1. System resource constraints on the server side
1.2. Active adaptation of application services on the server side
1.3. Active adaptation of the http client connection of the client
2. Design of client connection pool and thread pool
2.1. The "number of concurrent threads" of the "runtime" actually left for us
2.2. Design scheme of client connection pool and thread pool
2.2.1. Loop "http asynchronous request" in a single thread
1. Overall structure
1.1. System resource constraints on the server side
When analyzing and designing this case, first understand and clarify some basic concepts:
1.1.1. Single physical server
1.1.1.1. Basic concepts
For each computer, its hardware design stage determines its physical resources: CPU performance parameters, memory I/O, disk I/O, etc.
Application process: The "service", "module" and "handle" of the application that the operating system manages and schedules uniformly.
User mode of in-process threads: Threads defined and executed in user applications.
The system state of threads in the process: the CPU of the physical device and its operating system manage all "user state" threads and their scheduling in a unified manner.
How many physical CPUs the device has: that is, the number of slots in the Socket on the motherboard
The number of cores of a single physical CPU core: the core of the CPU
Commonly referred to as multiprocessor: refers to the "logical processor" of the device.
Number of threads that a logical processor can handle: A queue that can and can only handle 2 threads.
Unit of time: Each logical processor of the CPU can only process one thread Worker, that is, it can only do one job; a maximum of two worker threads are pushed into the stack, last in first out; in the stack, the last entry is finished Pop up after the job, and the second one at the bottom of the stack starts to enter the queue and pop up after processing.
How many threads can be executed concurrently by a single physical server:
Formula = total number of logical CPUs * 2
For example, the 8-core server in the above figure can have up to 16 concurrent threads.
var
/// <summary>◆本机Windows平台默认线程池最大的工作线程数:<para></para></summary>
MaxLimitWorkerThreadCount:Integer;
/// <summary>◆本机默认连接池每个服务器的最大tcp并发连接数:<para></para></summary>
MaxConnectionsPerServer: Integer;
/// <summary>◆本机最大的并发http请求任务数:<para></para></summary>
MaxIFutureTask: Integer;
var
/// <summary>◆全局未来函数完成Rest和本地的同步Json的数据管理:<para>TKey,TValue</para></summary>
FDictionary_IFuture:TDictionary<string,IFuture<string>>=nil;
var
/// <summary>◆平台默认的连接池的最大连接数的默认值:<para>适用win7、win10、win11客户端注册表默认值</para></summary>
FConnectsPoolMaxCount: Integer =10;
/// <summary>◆连接池中的THTTPClient连接对象字典:<para></para></summary>
FConnectsManager_ObjDict: TObjectDictionary<string,THTTPClient>=nil;
/// <summary>◆有了对象字典FConnectsManager_ObjDict可以转化为数组.TArray，可以不再单独设定数组的键值对:<para>备用，暂时不用</para></summary>
//FConnectsManager_ObjDictArray: TArray<TPair<string,THTTPClient>>=nil;//连接池中的THTTPClient连接对象的数组键值对
/// <summary>◆暂不使用，至少4个才可枚举，连接池中的http客户端的连接状态字典:<para>Status状态字符串:</para>TTaskStatus = (Created, WaitingToRun, Running, Completed, WaitingForChildren, Canceled, Exception)</summary>
FEnumConnectsStatus: TDictionary<string,string>=nil;
/// <summary>◆连接池中的连接执行http的异步执行结果的字典:<para>errStatus= 'Completed';创建时'Created'</para><para>键值对说明:aHTTPRequest.Name,IAsyncResultStatus</para></summary>
FAppHttpConntsAsyncResults:TDictionary<string,string>=nil;
MaxLimitWorkerThreadCount :=
TThreadPoolStats.Get( TThreadPool.Default ).MaxLimitWorkerThreadCount;
//: If the device is an 8-core logical processor*2=16; when running, the system state will automatically reduce 1 thread, and at least 1 can be reserved The worker thread is used by other jobs to keep the continuous switching between threads
//—in other words, the total number of concurrent requests that should always be defined is at most = MaxLimitWorkerThreadCount -1 = 8 cores of my machine*2 -1 =15
HttpFilter := THttp_Filters_HttpBaseProtocolFilter .Create; //: The maximum number of tcp concurrent connections per server on the Windows platform—filter—readable and writable
//HttpFilter.MaxConnectionsPerServer := 128;//:The range specified by http and tcp protocols [ 2...128 ]//: Readable and writable, operating system default = 6 ————————————————————————————————————————————————————————————————————————————————————, THAT IS INSTABILITY :
= HttpFilter.MaxConnectionsPerServer;
:
FConnectsPoolMaxCount := MaxLimitWorkerThreadCount;
// Assignment system global variable - maximum number of concurrent future task requests - take the smaller:
if MaxLimitWorkerThreadCount > MaxConnectionsPerServer then
MaxIFutureTask := MaxConnectionsPerServer
else MaxIFutureTask := MaxLimitWorkerThreadCount;
Run gpedit.msc Local User Group Policy Editor:
As above, the device:
The maximum number of worker threads in the default thread pool of the current platform: 16 The
maximum number of concurrent tcp connections per server in the default connection pool of the current platform: 6
The maximum number of concurrent HTTP request tasks on the current platform: 6
Scope of application: Both server equipment and client equipment are applicable.
1.1.1.2, the maximum number of tcp concurrent connections for each server in the platform default connection pool
As above, some friends may ask why the maximum number of concurrent tcp connections per server in the default connection pool of the current platform is designed to be 6.
Originally, the underlying TCP protocol was initially unlimited; later, with the development of html, the http protocol was born. With the application and promotion of its upper layer protocol http, the tcp standard began to be modified to meet the needs of http. Initially, after Microsoft's IE browser defeated Nescape, it adapted 4 http1.0 protocols before IE8; after that, 2 http1.1 protocols adapted; a total of 6:
Its upper limit is 10 in the registry editor (applicable to win7, Win10, Win11):
run regedit:
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings
Upper limit = 10, suitable for win7, win10, win11, that is to say, when your client application sends 10 http requests concurrently, there will be no "unsafe" event in the thread. This value of 10 is also the upper limit for synchronously refreshing the UI during concurrent pressure testing. If this value is exceeded, the UI interface cannot be refreshed in the synchronization function of the thread. The current thread must be rotated and paused before returning to the thread , that is: every 10 concurrent requests are executed, the interface is refreshed once, otherwise it cannot be refreshed.
After that, Microsoft made another adjustment after .NET Core 3.0, and the customization range was adjusted from [ 2...128 ] to [ 2...256 ]. Our common nose: the father of Delphi compiler, C#, TS, and the main implementer of the .Net strategy, Anders Helsberger, played an important role in formulating the standard. The above not only must be followed in the native App development field , but also in the Html front-end field . The mainstream browsers also consciously follow this "not standard" unspoken rule standard.
At the same time, the Http2.0 protocol is also supported, and the SocketsHttpHandler.EnableMultipleHttp2Connections property is added to indicate whether other HTTP/2 connections can be established on the same server when the maximum number of concurrent streams is reached on all existing connections.
See Microsoft official documentation for details:
HttpClientHandler.MaxConnectionsPerServer 属性 (System.Net.Http) | Microsoft Learn
SocketsHttpHandler.PooledConnectionIdleTimeout 属性 (System.Net.Http) | Microsoft Learn
SocketsHttpHandler 类 (System.Net.Http) | Microsoft Learn
1.1.1.2, connection idle timeout
Platform default = 15 minutes:
1.1.2. Load balancing and clustering of physical servers
When a single physical server can no longer safely and effectively carry the concurrent requests from different "clients", "load balancing" and server cluster technology are needed, which is our jargon "heap server".
As shown in the example above, the IP mask should be designed based on the principle of proximity to the "physical region" of the http request.
For example, 221.237.0.0/16 is adapted to the client request of "China Telecom" in Wuhou District, Chengdu.
1.2. Active adaptation of application services on the server side
// 配置“会话超时”的时间：
SessionOptions.SessionTimeout=15(分钟计量，操作系统默认=15)
ASession.SessionTimeOut:=20(可改写，默认范围[1,20])
// “会话超时”后，连接“断开”，但并未从服务器连接池中清除即“注销”：
// 配置活动但空闲的会话，将其从服务器连接池中注销的超时时间：
SessionOptions.LockSessionTimeout:=60*1000;//1分钟(毫秒计量，操作系统默认=不限制分钟数，一般3分钟)
本地组策略编辑器-计算机配置-管理模板-Windows组件-Internet Explorer-安全功能-AJAX
1.3. Active adaptation of the http client connection of the client
The default value of System.Net.URLClient timeout = 60 seconds = 1 minute, which matches the lock session timeout of 60*1000 on the server above:
TURLClient = class
public const
DefaultConnectionTimeout = 60000;
DefaultSendTimeout = 60000;
DefaultResponseTimeout = 60000;
2. Design of client connection pool and thread pool
As mentioned above, in fact, the operating system is not designed to limit the upper limit of your system tcp connection, but in fact, you are always limited at runtime.
Why is this?
The reason is that in order not to block the UI interface and provide customers with a better UE experience, our http concurrent requests will always be put into "user-mode threads" for design and operation.
As mentioned in 1.1.1.1 of this article, the maximum number of concurrent threads that can be executed by a single computer (formula = total number of logical CPUs * 2) is determined by the hardware itself, not by your code. For example, in the configuration of the example computer in 1.1 of this article, one client has an 8-core logical processor , with a maximum of 16 concurrent threads, and in fact, one logical processor must be reserved to handle when the priority of the user application puts your App When switching to the "background", this reserved logical processor can be well switched to other threads.
Then, the "number of concurrent threads" of the "runtime" actually left for us = 16 - 1*2 = 14.
Assuming that the server is "competent" for "concurrent requests" from the client, then. Client, how to design it?
2.1. The "number of concurrent threads" of the "runtime" actually left for us
Formula = (total number of computer logical processors - 1) * 2 .
2.2. Design scheme of client connection pool and thread pool
How do we make full use of "the number of concurrent threads that are actually left for us at 'runtime'"? !
2.2.1. Loop "http asynchronous request" in a single thread
2.2.1.1. Realize "asynchronous concurrent request" with "blocking" "http synchronous request" mode
"http synchronous request" is always a "blocking" communication, so, is it better to use "http asynchronous request" or "http synchronous request"? Each has its pros and cons.
The outstanding representative of "blocking" communication, the cross-platform Indy open source communication library, has been widely used in all walks of life around the world, especially IT software development, as early as HTML was still in its infancy.
Similar to Indy's "blocking" communication, the code is very comfortable to write. When multi-threading is nested, or when threads need to be queued, it is indirect and clear without sloppiness.
"Blocking" communication, as long as you put it into a "thread" to run a "communication" request, it will not "block". The general design and code writing are "queued" like this:
// ...
if aTThread1.Running then aTThread1.aHttpRequest1.Exec;
if aTThread1.Finished if aTThread2.Running then aTThread2.aHttpRequest2.Exec;
// ...
Or run "concurrently" like this:
// ...
if aTThread1.Running then aTThread1.aHttpRequest1.Exec;
if aTThread2.Running then aTThread2.aHttpRequest2.Exec;
// ... 2.2.1.2, or similarly in a single thread, run each "asynchronous request" "concurrently" with common code :
// ...事前，定义好异步请求的字典httpASyncRequests_TDictionary，然后，从连接池中随机调用一个空闲的aHttpClient连接，来发送请求：
if httpASyncRequests_TDictionary.Count >0 then
TThread.CreateAnonymousThread(
procedure var loop:Integer;
begin // IFutures_TDictionary
for loop:=0 to httpASyncRequests_TDictionary.Count-1 do
begin
try
try
if Length(httpASyncRequests_TDictionary.ToArray) > loop then
begin
if httpASyncRequests_TDictionary.ToArray[loop].Key<>'' then
begin
TThread.Synchronize(TThread.Current,procedure begin
aHttpClient :=
Extract1RandomIdleTHttpClient( ifExistsRandomIdleTHttpClient() );
//:从连接池中随机调用一个空闲的aHttpClient连接
end);
//...这里省略了不少代码————意在判断请求的资源在本地持久的状态
// ——以使得计时器能反复产生新的单线程，完成没有完成的Worker
aIRequest:= aHttpClient.GetRequest(
httpASyncRequests_TDictionary.ToArray[loop].Value as THttpClient );
aHTTPResponse :=
aHttpClient.Execute( aIRequest, nil,aIRequest.Headers ) as IHTTPResponse;
end;
end;
except
//...
end;
finally
if aHTTPResponse<>nil then
begin
errStatus := aHTTPResponse.ContentAsString( TEncoding.UTF8 );
TThread.Synchronize(TThread.Current,
procedure begin
LogMe( httpASyncRequests_TDictionary.ToArray[loop].Key.Name
+'执行结果Json——' + errStatus );
end);
end;
end;
if (loop<>0) then
begin
Application.ProcessMessages; //:不卡外层的UI——外层监听在主线程
Sleep(50);//:循环不要太快了————否则——最底层的tcp连接繁忙————也可分不同请求差异化配置
end;
end;
end;
).start;We know that the bottom layer of http is tcp, and tcp is very time-consuming when creating a connection.
For a THttpClient connection class instance, as long as its scope is Application, its life cycle can be designed as one of the following:
◆ "Timeout, it will be automatically destroyed from the connection pool";
◆"Destroyed completely by application exit".
It does not have to be designed to be "destroyed when it is used up"; as long as it is not destroyed, the Http client can be " reused " in the local " connection pool " to improve "network" efficiency.
2.2.1.3. How to identify different http client connections
2.2.2. Implement multi-threaded "asynchronous concurrent requests" in the cycle of timer polling status
In order to adapt at the same time: the needs of the native embedded browser "multi-tab page" or its "background js worker dedicated thread", we can also implement the "asynchronous concurrent request" of concurrent threads in a non-blocking way of "asynchronous request" .
type
/// <summary>全局继承类___实例化1个统一的TBasicAction的组件类TComponent<para></para>来统一调度全局计时器的OnTimer事件<para></para></summary>
TimersOnTimerDownloadDatabase = class(TBasicAction)
public
/// <summary>执行全局计时继承类___统一调度执行其OnTimer的目标任务：<para></para></summary>
procedure ExecuteTarget(Target: TObject);
end;
/// <summary>全局继承类的实例___TokenOnTimer计时器事件的调度处理器<para></para></summary>
var TimerExec_OnTimerDownloadDatabase : TimersOnTimerDownloadDatabase;
{ TimersOnTimerDownloadDatabase }
procedure TimersOnTimerDownloadDatabase.ExecuteTarget(Target: TObject);
begin
if Target = frmDownloadDatabase.FTimerWx then
begin
frmDownloadDatabase.FTimerWx.Interval:= 1000; //:1秒
frmDownloadDatabase.FuturePrepare_Basicdata( Target );
frmDownloadDatabase.FutureExec_Basicdata( Target );
if FAppHttpConntManager.ifAllIFutureComplete = true then
begin
frmDownloadDatabase.FTimerWx.Enabled := false;
LogMe( '外层监听循环已经停止了...' ); //:外层监听循环在主线程
end;
end;
end;
The design idea of this mode is also applicable to the background dedicated thread of Html5 + ES6; it is also applicable to the concurrent request design in "WeChat Mini Program" and "Alipay Mini Program".
For the es6 background dedicated thread, please refer to another blog of the author:
" Browser cross-tab communication BroadCast and ServiceWorker-serial 1 "
In this "verification example", the author has been verified:
Assume that a user has 23 salespersons, and each salesperson will have its own "Sales Details Report" and "Expense Details Report" every month to facilitate the accountant to check the correctness of the data in the "Sales Commission Report". Monthly, 23*2+1 = 47 reports will be generated. Among them, the data flow of the "Sales Details Report" is relatively large, detailing the sales details of each order and product; finally, they will be automatically generated separately These 47 reports can be exported as PDF or Excel reports to facilitate social media sharing such as WeChat:
At the same time, because the output of the above results involves a large amount of basic data (resources with relatively low update frequency: such as "employee data", "customer data", "product data", "last period cost data", etc.), in order to improve customer experience , it needs to be silently encrypted and compressed by base64 and downloaded to the local "persistence" in the background before the application is output. When these resources need to be referenced locally, they can be quickly and efficiently loaded into the memory.
If these statistical reports are online, click to download one by one, and then load the UI, it will be too slow, and the operation will be too cumbersome and complicated, so "concurrent http requests" shows its powerful advantages.
If you like it, please bookmark and like it, and encourage me to continue to write original technology and experience sharing: