Talk about the principle of database connection pool

 This time we take the method of technological evolution to talk about the technology emergence process and principle of database connection pool, as well as the most popular open source database connection pool jar package.

1. How did we perform database operations in the early days

       1. Principle: Generally speaking, the process of accessing a database by a java application is:

 　　① Load the database driver;

 　　② Establish a database connection through jdbc;

 　　③ Access the database and execute the sql statement;

 　　④ Disconnect the database connection.

       2. Code 

       // Query all users

 

[java]  view plain copy  

 

1. Public void FindAllUsers(){  
2.        //1. Load the sqlserver driver object  
3.        DriverManager.registerDriver(new SQLServerDriver());               
4.        //2, establish a database connection through JDBC  
5.        Connection con =DriverManager.getConnection("jdbc:sqlserver://192.168.2.6:1433;DatabaseName=customer", "sa", "123");              
6.        //3, create state  
7.        Statement state =con.createStatement();             
8.        //4. Query the database and return the result  
9.        ResultSet result =state.executeQuery("select * from users");             
10.        //5, output query results  
11.        while(result.next()){  
12.               System.out.println(result.getString("email"));  
13.        }              
14.        //6, disconnect the database connection  
15.        result.close();  
16.        state.close();  
17.        con.close();  
18.  }  

 

3. Analysis

       In the process of program development, there are many problems: First, a database connection must be established for each web request. Establishing a connection is a time-consuming activity that takes 0.05s to 1s each time, and the system also allocates memory resources. This time for one or several database operations may not feel much overhead to the system. But for today's web applications, especially large e-commerce sites, it is normal for hundreds or even thousands of people to be online at the same time. In this case, frequent database connection operations will inevitably occupy a lot of system resources, the response speed of the website will definitely decrease, and even cause the server to crash. Not alarmist, this is the technical bottleneck restricting the development of some e-commerce websites. Secondly, for each database connection, it must be disconnected after use. Otherwise, if the program fails to close due to an exception, it will cause a memory leak in the database system, and eventually the database will have to be restarted. In addition, this kind of development cannot control the number of connection objects created, and system resources will be allocated without consideration. If there are too many connections, it may also lead to memory leaks and server crashes.

       In the above user query case, if 1000 people visit at the same time, there will be continuous database connection and disconnection operations:

 

 

       Through the above analysis, we can see that "database connection" is a kind of scarce resource, in order to ensure the normal use of the website, it should be properly managed. In fact, after we query the database, if we do not close the connection, but temporarily store it, when others use it, we will give this connection to them. It avoids the time consumption of the operation of establishing a database connection and disconnection at one time. The principle is as follows:

2. Database connection pool evolved from technology

       It can be seen from the above analysis that the root of the problem lies in the inefficient management of database connection resources. We know that for shared resources, there is a very famous design pattern: resource pool (resource pool). This mode is to solve the problem caused by the frequent allocation and release of resources. In order to solve the above problems, the database connection pool technology can be used. The basic idea of ​​database connection pooling is to create a "buffer pool" for database connections. Put a certain number of connections in the buffer pool in advance. When you need to establish a database connection, you only need to take one out of the "buffer pool" and put it back after use. We can prevent the system from endlessly connecting to the database by setting the maximum number of connections in the connection pool. More importantly, we can monitor the number and usage of database connections through the management mechanism of the connection pool, providing a basis for system development, testing and performance adjustment.

       We try to develop a connection pool ourselves to provide database connection services for the above query business:

       ① Write a class to implement the DataSource interface

       ② Create 10 connections at a time in the class constructor and save the connections in the LinkedList

       ③ Implement getConnection to return a connection from LinkedList

       ④ Provide a method to put the connection back into the connection pool

 

       1. Connection pool code       

[java]  view plain copy  

 

1. public class MyDataSource implements DataSource {  
2.           //Linked list --- implement stack structure  
3.           privateLinkedList<Connection> dataSources = new LinkedList<Connection>();  
4.   
5.           //Initialize the number of connections  
6.           publicMyDataSource() {  
7.                  //Create 10 connections at once  
8.                  for(int i = 0; i < 10; i++) {  
9.                         try {  
10.                            //1. Load the sqlserver driver object  
11.                            DriverManager.registerDriver(new SQLServerDriver());  
12.                            //2, establish a database connection through JDBC  
13.                            Connection con =DriverManager.getConnection(  
14.                               "jdbc:sqlserver://192.168.2.6:1433;DatabaseName=customer", "sa", "123");  
15.                            //3. Add the connection to the connection pool  
16.                            dataSources.add(con);  
17.                         } catch (Exception e) {  
18.                            e.printStackTrace ();  
19.                         }  
20.                  }  
21.           }  
22.   
23.           @Override  
24.           publicConnection getConnection() throws SQLException {  
25.                  //Remove a connection from the connection pool  
26.                  finalConnection conn = dataSources.removeFirst();  // remove the first connection and return  
27.                  returnconn;  
28.           }  
29.   
30.           // put the connection back into the connection pool  
31.           publicvoid releaseConnection(Connection conn) {  
32.                  dataSources.add(conn);  
33.                  }  
34.    }  

 

       2. Use the connection pool to refactor our user query function       

[java]  view plain copy  

 

1. //Query all users  
2. Public void FindAllUsers(){  
3.        //1. Use the connection pool to establish a database connection  
4.        MyDataSource dataSource = new MyDataSource();  
5.        Connection conn =dataSource.getConnection();          
6.        //2, create state  
7.        Statement state =con.createStatement();             
8.        //3. Query the database and return the result  
9.        ResultSet result =state.executeQuery("select * from users");             
10.        //4, output the query result  
11.        while(result.next()){  
12.               System.out.println(result.getString("email"));  
13.        }              
14.        //5, disconnect the database connection  
15.        result.close();  
16.        state.close();  
17.        //6, return the database connection to the connection pool  
18.        dataSource.releaseConnection(conn);  
19.  }  

 

 

       This is the principle of the database connection pool, which greatly improves the utilization of database connections and reduces the overhead of memory throughput. During the development process, we don't need to care about the database connection anymore. Naturally, there is a database connection pool to help us deal with it. Don't worry this time. But the connection pool needs to consider more than that, let's see what other issues need to be considered.

3. There are more issues to consider in connection pooling

       1. Concurrency issues

       In order for the connection management service to have maximum versatility, a multi-threaded environment, that is, concurrency issues, must be considered. This problem is relatively easy to solve, because the java language itself provides support for concurrency management, and the synchronized keyword can be used to ensure that threads are synchronized. The method of use is to add the synchronized keyword directly in front of the class method, such as:

      publicsynchronized connection getconnection（）

 

       2. Multiple database servers and multiple users

       For large-scale enterprise-level applications, it is often necessary to connect to different databases at the same time (such as connecting to oracle and sybase). How to connect to different databases? The strategy we adopted is to design a connection pool management class that conforms to the singleton pattern, and read a resource file when the only instance of the connection pool management class is created, in which the resource file stores the URL addresses of multiple databases and other information. According to the information provided by the resource file, multiple instances of the connection pool class are created, each of which is a connection pool for a specific database. The connection pool management class instance takes a name for each connection pool instance, and manages different connection pools through different names.

       For the situation that multiple users access the same database with different names and passwords, it can also be handled through the resource file, that is, multiple database connection information with the same url address but different user names and passwords can be set in the resource file.

 

       3. Transaction processing

       We know that transactions are atomic. At this time, the operation of the database is required to comply with the "all-all-nothing" principle, that is, for a group of SQL statements, either all or nothing is done.

       In the java language, the connection class itself provides support for transactions, which can be achieved by setting the autocommit property of the connection to false and then explicitly calling the commit or rollback method. However, to efficiently perform connection multiplexing, a corresponding transaction support mechanism must be provided. It can be implemented by using an exclusive connection for each transaction, which can greatly reduce the complexity of transaction management.

 

       4. Allocation and release of connection pool

       The allocation and release of the connection pool has a great impact on the performance of the system. Reasonable allocation and release can improve the multiplexing of connections, thereby reducing the overhead of establishing new connections, and at the same time speeding up user access.

       The free pool can be used for connection management. That is, the connections that have been created but have not been allocated are stored in a free pool according to the creation time. Whenever a user requests a connection, the system first checks whether there are idle connections in the idle pool. If there is, assign the connection with the longest establishment time (implemented by the sequential storage of the container) to him (actually, first judge whether the connection is valid, if available, assign it to the user, if not, remove the connection from the free pool. Delete, and re-check whether there are still connections in the idle pool); if not, check whether the currently opened connection pool has reached the maximum number of connections (maxconn) allowed by the connection pool. If not, create a new connection, and if it has reached, wait a certain time (timeout). If a connection is released within the waiting time, the connection can be assigned to the waiting user, and if the waiting time exceeds the predetermined time timeout, a null value (null) will be returned. The system only counts the connections that have been allocated and in use, and returns them to the free pool when they are used up. For the status of idle connections, a special thread can be opened for timing detection, which will cost a certain amount of system overhead, but can ensure a faster response speed. It is also possible to take the method of not opening up a special thread, but only detecting before the allocation.

 

       5. Configuration and maintenance of connection pool

       How many connections should be placed in the connection pool for the best performance of the system? The system can control the connections in the connection pool by setting the minimum number of connections (minconn) and the maximum number of connections (maxconn). The minimum number of connections is the number of connections created by the connection pool at system startup. If too many are created, the system will start slowly, but the system will respond quickly after creation; if too few are created, the system will start quickly but respond slowly. In this way, you can set a smaller minimum number of connections during development, which will lead to faster development, and set a larger number when the system is actually used, because it will be faster for accessing clients. The maximum number of connections is the maximum number of connections allowed in the connection pool. The specific setting depends on the access volume of the system. You can find the best point through repeated tests.

       How to ensure the minimum number of connections in the connection pool? There are two strategies, dynamic and static. Dynamic means that the connection pool is checked at regular intervals. If the number of connections is found to be less than the minimum number of connections, a corresponding number of new connections will be added to ensure the normal operation of the connection pool. Static is to check when you find that there are not enough idle connections.

4. There are mature open source connection pools for us to use in actual development

       It is enough to understand the principle of connection pooling. There is no need to write everything from scratch, which will take a lot of time, and the performance and stability may not meet the requirements. In fact, there are already many popular third-party database connection pool jar packages with good performance for us to use. Such as:

       1. Apache commons-dbcp connection pool

        Download: http://commons.apache.org/proper/commons-dbcp/

 

       2.c3p0 database connection pool

        Download: http://sourceforge.net/projects/c3p0/