JAVA and C # database connection pooling principle
In the current development of the Internet, high concurrency into the mainstream, and that is the most critical part of database operations and access, in the current development of the Internet, ORM framework not poor, such as: .Net-Core of EFCore, SqlSugar, Dapper. The JAVA Spring-DataJpa (EntityManager), Mybatis, MybatisPlus etc.
But when it comes to actually ORM essentially operate lowest-level database access components: Ado.net, Jdbc
Today I come to chat connection pooling principle two database access
Speaking before Ado.net and jdbc connection pool of data, we first need to understand what is the database connection pool
Connect to the database server typically takes a long time of several steps. Physical channels must be established (such as sockets or named pipes) must be the initial handshake with the server is necessary to analyze the connection string information, must be authenticated by the server is connected, to run a check must be registered in the current transaction, and the like.
In fact, most applications use only one or a few different connection configurations. This means that during the execution of the application, many of the same connection will be repeatedly opened and closed. This is the performance cost of Cpu. In order to minimize the cost of connecting open, ADO.NET using optimization techniques known as connection pooling. The optimization techniques Java is jdbc connection pool.
In general, the process of Java applications that access the database are:
- Load database driver;
- Database connection is established via jdbc;
- Access to the database, execute sql statements;
- Disconnect from the database.
This is a common Tomcat database connection step of mapping and database connections Jdbc
The process of .Net Framwork / .Net Core applications that access the database is the four core subject by the .NET data provider:
1.Connection: connect to the database 2.Command: execute database commands 3.DataReader: 4.DataAdapter responsible for reading data from a data source: Contact responsible for data collection and database
This is the mapping database connection Ado.net
Ado.net：
Ado.net database connection step:
1.新建一个数据库连接字符串
string conStr = “Data Source=.;Initial Catalog=MySchoolDB;Integrated Security=True”;
2.引入命名空间：
using System.Data.SqlClient;
3.创建SqlConnection对象
SqlConnection conn = new SqlConnection(conStr);
4.打开连接：
conn.Open();
5.关闭连接：
conn.Close();
五、使用Command对象的步骤：
1.创建数据库连接
SqlConnection conn = new SqlConnection(conStr);
2.定义sql语句
string sql = “insert into Admin values(‘值’)”;
3.创建SqlCommand对象
SqlCommand cmd = new SqlCommand(conn,sql);
4.执行命令
cmd.ExecuteScalar();
We already know that at the time of connection, if in the case of a sudden surge in traffic moment, then the thread will open up more of database access, which is the basic connection this time has been insufficient to cope with high concurrent access of the high QPS
This time Mircosoft created a database provided by the Data Provider Connection Pooling --Ado.net connection pool: it allows applications to save the connection used in the connection pool to avoid every time to complete the establishment / closing complete the process connection.
Data Provider complete the process of establishing a connection upon receipt of a connection request is:
- Establishing a new connection to the connection pool (i.e., "logical connection"), and the establishment of the "logical connection" corresponding "physical connection." Establish a "logical connection" must be accompanied by the establishment of a "physical connection."
- Data Provider attached complete a closed procedure is to shut down "logical connection" corresponding "physical connection" and destroyed "logical connection."
- Destruction "logical connection" must be accompanied by close "physical connection", SqlConnection.Open () is requesting a connection Data Provider does not necessarily need to be completed to establish a complete process of connecting to the Data Provider, may simply removed from the connection pool available can;
- SqlConnection.Close () is a request to close a connection, Data Provider does not necessarily need to complete a full course of closing the connection, you may only need the connector can be released back to the connection pool.
Now I write a test code to test without using a connection pool of database connections effect: while I detect a Cpu consumption with windows performance counters
class Program
{
static void Main(string[] args)
{
SqlConnection con = new SqlConnection("server=.\\sqlexpress;database=zsw;pooling=true;trusted_connection=true;uid=sa;pwd=zsw158991626ZSW;");
for (int i = 0; i < 10; i++)
{
try
{
con.Open();
Console.WriteLine("开始连接数据库" + System.Threading.Thread.CurrentThread.Name);
System.Threading.Thread.Sleep(1000);
}
catch (Exception e) { Console.WriteLine(e.Message); }
finally
{
con.Close();
System.Threading.Thread.Sleep(1000);
}
}
Console.Read();
}
}
This time my code is on the database connection pool, and my connections only one, but when we get rid of Console.Readkey setting pooling = false at this time when my data connection takes up 10, due to my computer sqlserver performance testing does not open, but you can later go online Baidu try to view the number of connections
but! .Net Core connected to a database seems to be the default open data connection pool, which I find for a long document no results.
This pooling So what is it?
Whenever the program needs time to read and write to the database. Connection.Open () will use the ConnectionString to connect to the database, the database will establish a connection for the program, and remains open, after which the program can use T-SQL statements to query / update the database. When performing the Connection.Close (), the database will close the current connection. Well, everything looks are so methodical.
But if my program need not timed to open and close connections (such as ASP.Net or Web Service), for example, when Http Request sent to the server,, we need to open the Connection and then use the Select * from Table returns a DataTable / DataSet to the client / browser, and then close the current Connection. That every Open / Close Connection so frequent operation for the entire system will undoubtedly become a waste.
ADO.Net Team will give a better solution methods. The previous saved Connection, next time will need to open connections when the previous Connection to the next connection. This is the Connection Pool.
Well, this pooling is how to work it?
First, when a program is executed Connection.open (), ADO.net need to judge, this connection supports Connection Pool (Pooling default is True), if you specify False, ADO.net is created between the database and a connection (for to avoid confusion, all connected to the database, use the "connect" description), then returns to the program. If you specify True, ADO.net will be based ConnectString create a Connection Pool, and then fill in Connection to Connection Pool (all connections .net program, use the "Connection" description). Connection to fill the number determined by the Min Pool Size (default is 0) properties. For example, if 5 is specified, the time between ADO.net open SQL database 5 is connected, then the Connection 4, stored in the Connection Pool, a Connection returning.
When the program is executed to Connection.close () of. Pooling Connection Pool is put If True, ADO.net current Connection put and held between the connection database. While also determining Connection Lifetime (default is 0) properties, 0 represents infinity, if there is time exceeds Connection Connection LifeTime, ADO.net Connection will be closed while disconnected from the connection to the database, rather than save it back to Connection Pool in.
(This is primarily used in the SQL database clustering, load balancing purposes). If Pooling designated False, the disconnected directly between database.
Then the next time Connection.Open () implementation of the time, ADO.Net Connection will determine whether the new ConnectionString connectionString previously stored in the Connection Pool is consistent. (ADO.Net ConnectionString will turn into a binary stream, so to say, the new ConnectionString must exactly match the Connection's ConnectionString saved in Connection Pool in even added an extra spaces, or modify certain attributes in the Connection String the order will make ADO.Net think this is a new connection, and re-create a new connection, so if you are using a UserID, Password authentication method, modify the Password will lead to a connection, if you are using the SQL integrated authentication, you need to save two connections using the same).
ADO.net then necessary to determine whether the current Connection Pool Connection there may be used (not being used by another program), if not, it is necessary to determine ADO.net Max Pool Size ConnectionString setting (default is 100), if the Connection All connection Pool does not reach Max Pool Size, ADO.net will connect to the database again to create a connection, and then will return to the connection program.
If you have reached MaxPoolSize, ADO.net again will not create any new connections, but wait Connection Pool are occupied by other programs Connection release, this waiting time by SqlConnection.ConnectionTimeout (default is 15 seconds) restrictions, also that is, if more than 15 seconds, SqlConnection will throw out error (so sometimes if SqlConnection.open () method throws a timeout error, one possible reason is that there is no timely Connnection previous close, while the number of Connection Pool reached the MaxPoolSize.)
If one is available Connection, removed from the Connection Pool Connection is not directly returned to the program, ADO.net also need to check the ConnectionReset ConnectionString property (the default is True) the need for the most time Connection reset. This is because, before returning from the program Connection may have been modified, for example, use SqlConnection.ChangeDatabase method to modify the current connection, then returned Connection may have been connected to the current Connection String is not specified in the Initial Catalog database. It is necessary to reset a current connection. However, due to all the extra checks will increase ADO.net Connection Pool overhead on the system.
Connection pool is created for each unique connection string. When you create a pool, a plurality of connection object is created and added to the pool in order to meet the minimum pool size. The need to add to the pool is connected, but can not exceed the maximum pool size (default is 100). Released back into the pool when the connection is closed or open.
to sum up
When SqlConnection object request, if there is an available connection from the pool to obtain the object. Available to connect, it must not be used, having an associated transaction context matching or any transaction context, and the server has a valid link.
Connection pooling process by reallocating when a connection is released back to the pool, to meet these connection requests. If the maximum pool size has been reached and there is no available connection, the request will be queued. Then, the process of trying to re-establish the pool any connection until reaching the timeout (default is 15 seconds). If the pool before the connection process can not meet the request, an exception is thrown.
Good use of connection pooling will greatly improve the performance of your application. On the contrary, if used improperly, then more harm than benefit. In general, it should follow the following principles:
- In the latest application for connection time, the connection is released at the earliest time.
- Turn off the relevant user-defined transaction when the connection is closed.
- Secure and maintain the connection pool has at least one open connection.
- Try to avoid generating cell debris. Including cell debris generated and integrated security pool debris generated by the use of many databases.
Jdbc：
Having a Ado.net come talk Jdbc, because I also recently learned Java language so naturally the two were compared
The default JDBC database connection pool
The method does not provide a connection pool JDBC API. Some large WEB application servers such as BEA's WebLogic and IBM's WebSphere provide connection pooling mechanism, but there must be a special class methods of its third-party support for the connection pool usage.
JDBC database connection pool used to represent javax.sql.DataSource, the DataSource only an interface that is typically provided by a server implemented (Weblogic, WebSphere, Tomcat), there are provided a number of open source implementation:
①DBCP database connection pool
②C3P0 database connection pool
DataSource is usually referred to as data source, connection pooling and comprising two portions connected pool management, the habits often referred to as a connection pool DataSource
Connected to different data sources and databases, without creating a plurality of data sources, which is connected to the database generating plant, the entire application needs to only one data source.
After the end of database access, the program is still the same as before close the database connection: conn.close (); but above code does not close the physical connection to the database, it is only the database connection release, returned to the database connection pool.
JDBC database connection pool mechanism:
Database connection pool is responsible for the distribution, management and release of database connections. Database connection pool at initialization, creates a certain number of connections into the connection pool, the number of these database connection is made to set the minimum number of database connections. Whether they have not used the database connection, the connection pool will always remain there, at least so many number of connections. The maximum number of database connection pool limits the maximum number of connections occupied by the connection pool, when the number of application requests connection to the connection pool is greater than this limit, the request will be added to the waiting queue. The minimum number of connections and the maximum number of database connections set to consider several factors:
1) The minimum number of connections to the database connection pool of database connections will remain, if when an application uses the database connection is not particularly large, there will be a large number of database connection resources are wasted;
2) maximum number of connections is the maximum number of connections to the database application can, if this number exceeds a database connection requests, database connection request will later be added to the waiting queue, so that the latter will affect the operation of the database;
3) If the minimum number of connections and the maximum number of connections that much difference, then the connection request will be the first to profit, after more than a minimum number of connections the connection is equivalent to re-create a new database connection. However, these larger than the minimum number of connections is not finished using the database connection is immediately released, it will be placed in the connection pool after waiting for the idle timeout reused or released.
Now we try to use DBCP way to connect to the database
1, first create a maven project and create a new file db.properties in resources
jdbc.driver=com.mysql.cj.jdbc.Driver
jdbc.url=jdbc:mysql://localhost:3306/mysql?&useSSL=false&serverTimezone=UTC
jdbc.username=root //用户名
jdbc.password=123456 //密码
initSize=10 //初始化连接数
maxTotal=200 //最大连接数
maxIdle=60 //最大空闲数，数据库连接的最大空闲时间。超过空闲时间，数据库连接将被标记为不可用，然后被释放。设为0表示无限制。
2, followed by introduction of the package dependency maven
<dependencies>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.19</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-dbcp2</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.2</version>
</dependency>
</dependencies>
Direct copy and paste, but please note that you must re-jdk default version 8 or higher!
3, then a new class JdbcUtil
package com.jdbc.util;
import org.apache.commons.dbcp2.BasicDataSource;
import java.io.InputStream;
import java.sql.Connection;
import java.util.Properties;
/**
* DBCP的方式链接数据库
*/
public class JdbcUtil {
private static String driver;
private static String url;
private static String username;
private static String password;
private static int initSize;
private static int maxTotal;
private static int maxIdle;
private static BasicDataSource ds;
static {
ds = new BasicDataSource();
Properties cfg=new Properties();
try { //读取db.properties文件
InputStream in = JdbcUtil.class
.getClassLoader()
.getResourceAsStream("db.properties");
cfg.load(in);
//初始化参数
driver=cfg.getProperty("jdbc.driver");
url=cfg.getProperty("jdbc.url");
username=cfg.getProperty("jdbc.username");
password=cfg.getProperty("jdbc.password");
initSize=Integer.parseInt(cfg.getProperty("initSize"));
maxTotal=Integer.parseInt(cfg.getProperty("maxTotal"));
maxIdle=Integer.parseInt(cfg.getProperty("maxIdle"));
in.close();
//初始化连接池
ds.setDriverClassName(driver);
ds.setUrl(url);
ds.setUsername(username);
ds.setPassword(password);
ds.setInitialSize(initSize);
ds.setMaxTotal(maxTotal);
ds.setMaxIdle(maxIdle);
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
public static Connection getConnection() {//连接数据库封装类
try {
/*
* getConnection()从连接池中获取的重用
* 连接，如果连接池满了，则等待。
* 如果有归还的连接线，则获取重用的连接
*/
Connection conn = ds.getConnection();
return conn;
} catch (Exception e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
public static void close(Connection conn) {//关闭数据库的连接方法，封装复杂的关闭过程；
if(conn!=null) {
try {
//将用过的连接归还到连接池
conn.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
4, we write a class test to verify
package com.jdbc.service;
import com.jdbc.util.JdbcUtil;
import java.sql.Connection;
import java.sql.SQLException;
public class JdbcTest {
public static void main(String[] args) {
try {
for (int i=0;i<1000;i++){
Thread a= new Thread(new TestThread(),"线程:"+(i+1));
a.start();
System.out.println(a.getName()+"已启动");
}
}
catch (Exception ex){
ex.printStackTrace();
}
}
private static class TestThread implements Runnable{
private Connection con= JdbcUtil.getConnection();
@Override
public void run() {
try {
if (con.isClosed()){
System.out.println("连接已经关闭");
}
} catch (SQLException e) {
e.printStackTrace();
}
finally {
//JdbcUtil.close(con);
//System.out.println("\t"+Thread.currentThread().getName()+"已关闭");
}
}
}
}
Now run the test and found that output
Port occupies 200, opened the thread pool work, but I did not release the connection port, but I changed a bit db.properties maximum number of connections was 300, now let's look at the results
You can see our database connection is being given, this is why? Because my local MySQL connection port 200 only when the port is connected more than 200 will collapse. It is also common database connectivity bottleneck
Now we close the connection uncommented code, you can see that even if there are 1,000 connections will be performed quickly, and will not take up extra port
DBCP way is the way of the use of Tomcat servers, so in tomcat using a data connection pool or necessary, at least in general concurrency Go On! The database connection pool can use either integrated with the application server, the application can also be used independently.
to sum up
The JDBC JAVA and Microsoft Ado.net in fact essential difference is not large, as are for the operation of the database, the largest database of its fundamental performance bottleneck link really is the problem?
Then the index of the database, the database now sub-library sub-table, there is a separation technology to read and write is because of what? Therefore, the database connection pool is one of the performance optimization, database optimization and many more in the future operation of waiting for people to explore
Such as the current Alibaba Druid, is the most demand better results, Microsoft's ORM have also opened early are optimized for database connection pooling, which marks the future of the Internet is not a performance bottleneck has been the situation with a traditional relational database
Future Nosql popular intervention so that high concurrency better to assume the task of large Internet projects!
In fact, for Ado.net and jdbc I did not take the time to compare performance, I like C # also like Java, this is an excellent language learn from each other, and then we write the code, as learning algorithm, if you start to understand how to write I believe a good code, you will not care about the performance advantages of the language.
This article refers to:
https://blog.csdn.net/huwei2003/article/details/71459198
https://blog.csdn.net/hliq5399/article/details/73292023
https://blog.csdn.net/weixin_40751299/article/details/81609332
https://www.cnblogs.com/justdoitba/p/8087984.html
https://www.cnblogs.com/albertrui/p/8421791.html
https://blog.csdn.net/L_it123/article/details/88205528
Thanks above bigwigs of the article, so I can save time to write this article.