Preface

In the previous article , we introduced the background of elastic database connection failure and discussed the related content of HikariCP connection pool detection strategy. In this article, we will continue to explore another commonly used online connection pool - Druid, and introduce you to how to implement the best-practice elastic database connection pool exploration strategy when using Druid.

Druid

The version of Druid is iteratively updated quickly, and there are also many parameters to explore and configure. This means that even the same parameters may achieve different effects in different versions. However, the logic implementation related to live detection only exists in two functions in the source code. Let's first list the parameters related to Druid live detection, and then take a detailed look at the use of these parameters by the source code implementation. In the future, if we encounter a situation where the configured probing does not take effect during development, we can look at the source code of the corresponding version to determine whether our probing is configured correctly.

The following are the parameters related to Druid exploration:

parameter name	illustrate	Defaults
initialSize	The number of physical connections established during initialization. Initialization occurs when the init method is explicitly called, or when getConnection is first used.	0
minIdle	Minimum number of connection pools.	0
maxActive	Maximum number of connection pools.	8
testOnBorrow	When applying for a connection, execute the SQL configured by validationQuery to check whether the connection is valid. Doing this configuration will reduce performance.	false
testOnReturn	When returning the connection, execute validationQuery to check whether the connection is valid. This configuration will reduce performance.	false
testWhileIdle	It is recommended to configure it to true, which will not affect performance and ensure security. Detect when applying for a connection in the connection pool. If the idle time is greater than timeBetweenEvictionRunsMillis, execute validationQuery to check whether the connection is valid.	True for most versions
timeBetweenEvictionRunsMillis	1) The Destroy thread will detect the connection interval and execute a DestroyTask every this value. 2) For the judgment basis of testWhileIdle, please see the description of testWhileIdle attribute for details.	Most versions are 1 minute
keepAlive	For connections within the number of minIdles in the connection pool, if the idle time exceeds minEvictableIdleTimeMillis, a live detection operation will be performed. This parameter is only supported in versions 1.0.28 or above for detailed instructions. Please refer to the official documentation.	false
keepAliveBetweenTimeMillis	Use with keepAlive is not supported in lower versions. If the idle time is less than timeBetweenEvictionRunsMillis but greater than keepAliveBetweenTimeMillis, the live detection operation will be executed.	Most versions are 2 minutes
validationQuery	The sql used to check whether the connection is valid requires a query statement.	select 1
validationQueryTimeout	Unit: seconds, timeout for detecting whether the connection is valid. The underlying method calls the void setQueryTimeout(int seconds) method of the jdbc Statement object
minEvictableIdleTimeMills	When the connection idle time is greater than this value, close the connection whose idle connection is greater than minIdle, similar to hikaricp's idleTimeout	30 minutes
maxEvictableIdleTimeMillis	When the connection idle time is greater than this value, the connection will be closed regardless of minIdle, similar to hikaricp's maxlifetime (lower versions do not support this)	7 hours

Druid's exploration is mainly implemented by the following two functions:

com.alibaba.druid.pool.DruidDataSource#getConnectionDirect

getConnectionDirect is a function that is called every time a connection is taken from the connection pool. We can see from the code below that if testOnBorrow is true, whether the connection is valid will be checked before each connection is obtained. If testOnBorrow is false and testWhileIdle is true, you need to determine whether the idle time of the connection exceeds the value set by timeBetweenEvictionRunsMillis. If it exceeds, perform live detection. Expired connections will be discarded and the number of minIdles added to the connection pool will be added. The default value of timeBetweenEvictionRunsMillis in most versions is 1 minute. As long as this value is set for less than ten minutes and testWhileIdle is turned on, it is guaranteed that the failed connection closed by the gateway will not be obtained.

In earlier versions that do not support keepalive, you can only rely on testOnBorrow or testWhileIdle for detection. It is recommended to configure testWhileIdle for testing. In high-concurrency scenarios, the performance consumption of this method will be smaller.

com.alibaba.druid.pool.DruidDataSource#shrink(boolean, boolean)

In the code below we can see that the shrink method is called in the run method of the DestroyTask thread to destroy the connection in the connection pool. If timeBetweenEvictionRunsMillis is greater than 0, the destroyTask.run(boolean, boolean) method will be called every this time interval, that is, the shrink method will be executed.

As can be seen from the above code, the shrink method uses the keepAlive parameter. It should be noted that in different versions of Druid, the support and implementation logic of the keepAlive parameter may be different. The official recommendation is that when using the keepAlive parameter, you should use version 1.1.21 or above. Although the official documentation states that if the idle time exceeds minEvictableIdleTimeMillis, the live detection operation will be performed, but in higher versions, the execution time of this live detection operation is also affected by the keepAliveBetweenTimeMillis parameter. Therefore, in higher versions, if you want to use the keepAlive parameter correctly, you need to understand its implementation logic in the specific version.

The following code is a comparison of the source code of the shrink method in versions 1.1.10 and 1.1.21:

First, take a look at the source code of version 1.1.10. It will first determine whether the connection idle time is greater than minEvictableIdleTimeMillis. If so, then proceed to the second step of judgment: whether there are more idle connections than minIdle. If so, these connections are added to the array of evicted connections for subsequent eviction operations. If not, judge again whether the connection idle time is greater than maxEvictableIdleTimeMillis. If so, add these connections to the array of evicted connections. If not, make the final judgment: whether the keepAlive configuration is enabled. If enabled, these connections will be added to the keep-alive connection array for subsequent live detection operations.

In version 1.1.21, the overall logic of the shrink method is similar to that of version 1.1.10, but a new parameter named keepAliveBetweenTimeMillis is added. This parameter determines the time interval for active detection using keepAlive. Its default value is 2 minutes. If keepalive is turned on and the idle time is greater than this value, active detection will be performed.

Another difference is that when performing live detection operations, version 1.1.10 will only close invalid connections, but version 1.1.21 goes one step further. In addition to closing invalid connections, it will also automatically add connections to reach the minimum connection of minIdle. number.

1.1.10 1.1.21

In summary, druid's liveness detection parameters did not have a scheduled liveness detection function before version 1.0.28. It can only detect whether it is valid before getting a connection each time. It is recommended to configure testWhileIdle to true, which will not affect performance too much in high concurrency situations. If Those with high availability requirements can turn on testOnBorrow to check the validity of the connection every time it obtains a connection. In higher versions, you can use the keepAlive parameter to keep the connection alive. For applications that use Druid connection pool online, it is recommended to use version 1.1.21 or higher that supports keepAlive.

JED configuration template:

Druid1.1.10

<propertyname="testWhileIdle"value="true"/>    
<propertyname="validationQuery"value="SELECT 1"/>    
<propertyname="timeBetweenEvictionRunsMillis"value="30000"/>    
<propertyname="minEvictableIdleTimeMillis"value="300000"/>
<propertyname="keepAlive"value=true/>

This version supports keepAlive and can configure the minEvictableIdleTimeMillis time to be less than 10 minutes, which can efficiently detect liveness and prevent the gateway from closing the connection.

Druid1.1.9

Same as 1.1.10

Druid1.0.9

<propertyname="testWhileIdle"value="true"/>    
<propertyname="validationQuery"value="SELECT 1"/>    
<propertyname="timeBetweenEvictionRunsMillis"value="30000"/>    
<propertyname="minEvictableIdleTimeMillis"value="300000"/>

This version does not support keepAlive, which can only be detected when obtaining the connection object. For high availability, testOnBorrow can also be turned on.

Author: JD Retail Wang Leixin

Source: JD Cloud Developer Community Please indicate the source when reprinting

Research on elastic database connection pool exploration strategies (2) - Druid | JD Cloud technical team