How to configure the number of database connection pools and HikariCP

HikariCP connection pool

HikariCP connection pool is a high-performance JDBC connection pool, with three characteristics marked on the official website: fast, simple, and reliable, and its performance is better than other connection pools.

The official website explains in detail some optimizations made by HikariCP, which are summarized as follows:

Bytecode streamlining: optimize the code until the compiled bytecode is the least (flattening the inheritance hierarchy, concealing member variables, eliminating forced type conversion), so that the CPU cache can load more program code;
Optimize proxies and interceptors: reduce code. For example, HikariCP's Statement proxy has only 100 lines of code, which is only one-tenth of BoneCP;
Custom array type (FastStatementList) instead of ArrayList: avoid range check every time get() is called, and avoid scanning from start to end when calling remove();
Custom lock-free collection type (ConcurrentBag): Improve the efficiency of concurrent reading and writing;

Use of HikariCP

Since SpringBoot2 integrates HikariCP by default, dependencies need to be introduced again. Mainly depends on the project mysql and springboot mybatis.

Yaml configuration

# 配置数据源信息
spring:
  datasource:                                           # 数据源的相关配置
    type: com.zaxxer.hikari.HikariDataSource          # 数据源类型：HikariCP
    driver-class-name: com.mysql.jdbc.Driver          # mysql驱动
    url: jdbc:mysql://localhost:3306/foodie-dev?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true
    username: root
    password: 123456
    hikari:
      connection-timeout: 30000        # 等待连接池分配连接的最大时长（毫秒），超过这个时长还没可用的连接则发生SQLException， 默认:30秒
      minimum-idle: 5                  # 最小连接数
      maximum-pool-size: 20            # 最大连接数
      auto-commit: true                # 事务自动提交
      idle-timeout: 600000             # 连接超时的最大时长（毫秒），超时则被释放（retired），默认:10分钟
      pool-name: DateSourceHikariCP     # 连接池名字
      max-lifetime: 1800000             # 连接的生命时长（毫秒），超时而且没被使用则被释放（retired），默认:30分钟 1800000ms
      connection-test-query: SELECT 1  # 连接测试语句

# mybatis配置
mybatis:
  type-aliases-package: com.lzp.pojo          # 所有POJO类所在包路径
  mapper-locations: classpath:mapper/*.xml      # mapper映射文件

Setting the number of connections in the connection pool

Introduce database connection pool

Database connection is a limited and expensive resource. A database connection object corresponds to a connection to a physical database. If a new connection is created for each database operation and released after use, the system performance will be low, which leads to the connection pool concept.

The database connection pool is responsible for allocating, managing and releasing database connections. It allows applications to reuse an existing database connection and can be regarded as a container for storing database connections.

The database connection pool adopts a resource pool design mode for resource sharing, avoiding frequent resource allocation and release problems. At the same time, it is convenient for unified management. Through the control of the connection pool, the connection between the system and the database can be restricted, and the number of connections and usage of the database can be monitored.

The necessity of connection pool

1. Do not use the connection pool process

Let's take access to MySQL as an example to execute a SQL command. If the connection pool is not used, what processes need to go through.

Steps not to use the database connection pool:

Three-way handshake for TCP connection establishment
Three-way handshake for MySQL authentication
Real SQL execution
MySQL shutdown
TCP's four-way handshake is closed

It can be seen that in order to execute a piece of SQL, there are a lot of network interactions.
Advantages: Simple to implement
Disadvantages:

More network IO
The database load is high
Long response time and low QPS
Applications frequently create and close connections, resulting in more temporary objects and frequent GC
After closing the connection, there will be a lot of TIME_WAIT TCP status (closed after 2 MSL)

2. Use the connection pool process

Steps to use database connection pool:

The first time you visit, you need to establish a connection. However, the subsequent visits will reuse the previously created connection and execute the SQL statement directly.
advantage:

Less network overhead
The performance of the system will be substantially improved
No trouble TIME_WAIT state

Database connection number setting

The system can control the connections in the connection pool by setting parameters such as the minimum number of connections and the maximum number of connections. The minimum number of connections is the number of database connections created when the system starts. The minimum number of connections is small, the startup is fast and the response is slow. Usually the setting is larger. The minimum number of connections can be set to 5-10. The maximum number of connections is set according to the hardware configuration. The 4-core machine can be set to 10, and the 8-core machine can be set to 20.

The default maximum and minimum number of connections for HikariCP is 10. The author's suggestion is to set the maximum and minimum number of connections to the same value and maintain a high-performance connection pool.

Why is the number of connection pools not as large as possible?

The first point, first of all, we need to know that a single-core CPU "simultaneously" runs multiple threads, which is just an illusion. A single-core CPU can only execute one thread at a time, and then the operating system switches context, the CPU core quickly schedules, and executes the code of another thread. This involves a lot of extra performance loss caused by context switching.

The second point, from the above, an N-core server can provide optimal performance by setting the number of database connections to N. However, the actual situation will be affected by disk IO and network IO. During the IO waiting time, the thread is blocked and waiting, and the CPU is in an idle state. Therefore, when threads are processing I/O-intensive business operations, it is necessary to set the number of threads/connections to be larger than that of the CPU to improve throughput.

Calculation formula for the number of connections

连接数 = ((核心数 * 2) + 有效磁盘数)

The server CPU is a 4-core i7, and the connection pool size should be ((4 * 2) + 1) = 9 ~ 10. Specific needs to be adjusted according to actual business scenarios.

Business scene

For concurrent access, you can use a small database connection pool, and then put the remaining business threads in the queue to wait.
If long and short transactions are mixed in the system, the correct approach should be to create two connection pools, one for long transactions and one for "real-time" queries, that is, short transactions.

Reference documents

https://www.jianshu.com/p/15b846107a7c
https://www.cnblogs.com/cocoxu1992/p/11031908.html
https://blog.csdn.net/weiwosuoai/article/details/89955003