database connection pool
The database connection pool is responsible for allocating, managing and releasing database connections, which allows an application to reuse an existing database connection instead of re-establishing one; release the database connection whose idle time exceeds the maximum idle time to avoid failure because the database connection is not released Caused by missing database connections. This technology can significantly improve the performance of database operations.
Influencing factors
When the database connection pool is initialized, a certain number of database connections will be created and placed in the connection pool. The number of these database connections is restricted by the minimum number of database connections. Regardless of whether these database connections are used or not, the connection pool will always be guaranteed to have at least this many connections. The maximum number of database connections in the connection pool limits the maximum number of connections that the connection pool can occupy. When the number of connections requested by the application from the connection pool exceeds the maximum number of connections, these requests will be added to the waiting queue. The setting of the minimum number of connections and the maximum number of connections in the database connection pool should take into account the following factors:
- The minimum number of connections is the database connection kept by the connection pool, so if the application does not use a lot of database connections, a large amount of database connection resources will be wasted.
- The maximum number of connections is the maximum number of connections that the connection pool can apply for. If the database connection request exceeds this number, subsequent database connection requests will be added to the waiting queue, which will affect subsequent database operations.
- The gap between the minimum number of connections and the maximum number of connections If the difference between the minimum number of connections and the maximum number of connections is too large, the first connection request will be profitable, and subsequent connection requests exceeding the minimum number of connections are equivalent to establishing a new database connection. However, these database connections that are greater than the minimum number of connections will not be released immediately after use, and will be placed in the connection pool for reuse or released after an idle timeout.
principle
The basic idea of the connection pool is to store the database connection as an object in the memory when the system is initialized. When the user needs to access the database, instead of establishing a new connection, an established idle connection object is taken from the connection pool. . After use, the user does not close the connection, but puts the connection back into the connection pool for the next request to access. The establishment and disconnection of connections are managed by the connection pool itself. At the same time, you can also control the initial number of connections in the connection pool, the upper and lower limits of connections, the maximum number of times each connection is used, the maximum idle time, etc. by setting the parameters of the connection pool. You can also monitor the number of database connections, usage, etc. through its own management mechanism.
Advantage
Performance improvements: The creation and destruction of database connections is an expensive operation. Connection pooling avoids frequent creation and destruction of connections by pre-creating a certain number of database connections and keeping them in the pool. This can significantly reduce the time required for each request and greatly improve the performance of the system.
Resource management: The connection pool can effectively manage and allocate database connections. It limits the number of connections to prevent resource overuse and thus avoid overloading the database server. Connection pooling can also handle connection reuse, reducing the overhead of frequently creating new connections.
Scalability: With connection pooling, you can adjust the size of the connection pool according to your needs. This allows the system to automatically expand or contract the number of connections as load changes to meet different needs. This scalability enables the system to better cope with high concurrency and high traffic situations.
Connection reuse: The connection pool can reuse connections instead of creating new connections every time. This avoids the overhead of frequently establishing and disconnecting connections and reduces communication overhead with the database server. By multiplexing connections, system resources can be used more efficiently.
Error handling: The connection pool can provide an error handling mechanism to handle database connection errors. It monitors the state of the connection and recreates or repairs the connection if an error occurs. In this way, the connection pool can improve the stability and reliability of the system.
Python implementation
dbutils.pooled_db is a library in Python that provides an implementation of a database connection pool. This library is widely used to manage and maintain database connection pools to improve the performance of applications interacting with databases.
import pymysql
from dbutils.pooled_db import PooledDB
host = 'localhost'
port = 3306
user = 'root'
password = '123456'
database = 'evos_gr_version1'
class MySQLConnectionPool:
def __init__(self,):
self.pool = PooledDB(
creator=pymysql, # 使用链接数据库的模块
mincached=10, # 初始化时，链接池中至少创建的链接，0表示不创建
maxconnections=200, # 连接池允许的最大连接数，0和None表示不限制连接数
blocking=True, # 连接池中如果没有可用连接后，是否阻塞等待。True，等待；False，不等待然后报错
host=host,
port=port,
user=user,
password=password,
database=database
)
def open(self):
self.conn = self.pool.connection()
self.cursor = self.conn.cursor(cursor=pymysql.cursors.DictCursor) # 表示读取的数据为字典类型
return self.conn, self.cursor
def close(self, cursor, conn):
cursor.close()
conn.close()
def select_one(self, sql, *args):
"""查询单条数据"""
conn, cursor = self.open()
cursor.execute(sql, args)
result = cursor.fetchone()
self.close(conn, cursor)
return result
def select_all(self, sql, *args):
"""查询多条数据"""
conn, cursor = self.open()
cursor.execute(sql, args)
result = cursor.fetchall()
self.close(conn, cursor)
return result
def insert_one(self, sql, *args):
"""插入单条数据"""
try:
self.execute(sql, args, isNeed=True)
except Exception as err:
return {
'result': False, 'err': err}
def insert_all(self, sql, datas):
"""插入多条批量插入"""
conn, cursor = self.open()
try:
cursor.executemany(sql, datas)
conn.commit()
return {
'result': True, 'id': int(cursor.lastrowid)}
except Exception as err:
conn.rollback()
return {
'result': False, 'err': err}
def update_one(self, sql, args):
"""更新数据"""
self.execute(sql, args, isNeed=True)
def delete_one(self, sql, *args):
"""删除数据"""
self.execute(sql, args, isNeed=True)
def execute(self, sql, args, isNeed=False):
"""
执行
:param isNeed 是否需要回滚
"""
conn, cursor = self.open()
if isNeed:
try:
cursor.execute(sql, args)
conn.commit()
except:
conn.rollback()
else:
cursor.execute(sql, args)
conn.commit()
self.close(conn, cursor)
connect
mysql = MySQLConnectionPool()
1. insert
sql_insert_one = "insert into `configuration` (`system`, blocksize, inodesize) values (%s,%s,%s)"
mysql.insert_one(sql_insert_one, ('ext4', '1024', 512))
datas = [
('ext4', '1024', 512),
('ext2', '1024', 256),
]
sql_insert_all = "insert into `configuration` (`system`, blocksize, inodesize) values (%s,%s,%s)"
mysql.insert_all(sql_insert_all, datas)
2. delete
sql_delete_one = "delete from `cofiguration` where `system`=%s "
mysql.delete_one(sql_delete_one, ('ext2',))
3. Modify
sql_update_one = "update `cofiguration` set blocksize=%s where `system`=%s"
mysql.update_one(sql_update_one, (1024, 'ext4'))
4. Query
sql_select_one = "select * from `cofiguration` where `system`=%s"
results = mysql.select_one(sql_select_one, ('ext4',))
print(results)
sql_select_all = "select * from `cofiguration` where `system`=%s"
results = mysql.select_all(sql_select_all, ('ext4',))
print(results)