Novice web development simple summary (11)-related optimization of database connection (data source DataSource)

table of Contents

Preface

One what is DataSource

Two common three implementation methods

1. Basic realization

2. Database connection pool implementation

（1）DBCP

（2）C3P0

(3) HikariCP

（4）Druid

3. Distributed implementation

Three summary 

---

Preface

A simple summary of the novice web development in Xiaobai (8)-Database HSQLDB instance This example, because it did not pay much attention to the connection and release of the database, just to be able to briefly understand the process of web application development. This time it is specifically summarized for this part.

The following code is a simple summary of the novice web development in Xiaobai (eight)-database HSQLDB instance this example, a code to connect to the database:

public Statement createStatement() {
        try {
            Class.forName("org.hsqldb.jdbcDriver");
        } catch (ClassNotFoundException e) {
            System.out.println("jdbcDriver not found!!!");
            return null;
        }
        try {
            //JDBC不支持自增,创建或连接数据库
            connection = DriverManager.getConnection(jdbcUrl, jdbcUser, jdbcPassword);
            if (connection == null) {
                return null;
            }
            //创建里面的表
            Statement statement = connection.createStatement();
            return statement;
        } catch (SQLException throwables) {
            throwables.printStackTrace();
        }
        return null;
    }

What is used here is DriverMananger's getConnection() to establish a connection with the database, but the connection with the database is originally a very resource-consuming operation and will generate a lot of system overhead. So there is a connection pool for the database. I think this is the same as the concept of threads and thread pools in Java. The thread pool was born to reduce the frequent creation and destruction of threads. Then the database connection pool is to reduce the frequent creation and disconnection of the database. And created.

One what is DataSource

When a web application accesses a database, it does not directly access the database, but accesses the database through a database driver, which is generally provided by the database manufacturer. Therefore, a JDBC interface is provided in Java EE to access the driver of the database, thereby accessing the database.

DataSource is a database connection container, responsible for managing the connection of the created database. When the web application is accessing the database, it does not need to connect to the database through the DriverManager, but obtains the Connection object directly from the DataSource, and then connects to the database.

It needs to be used with the database driver. There are usually three implementation forms:

• (1) Basic implementation: Just generate a standard Connection object, and when it is accessed, a new Connection will be generated, which does not support connection pooling and distributed;
• (2) Database connection pool implementation: generate Connection objects that automatically participate in connection pools, support connection pool connections, but do not support distributed;
• (3) Distributed implementation: Generate a Connection object, which can be used for distributed transactions, most of which participate in connection pool connections.

No matter what kind of DataSource, some configuration needs to be done: basic configuration, key configuration, performance configuration, etc.:

The basic configuration mainly includes the url, username, password, and driverClass to access the database;

The key configuration mainly includes the minimum number of connections, the number of initial connections, the maximum number of connections, and the maximum waiting time; (not required for the first implementation method)

Performance configuration mainly includes pre-caching settings, connection validity detection, timeout connection closure settings, etc.

Two common three implementation methods

The following is a simple summary of each application, and optimized on the example mentioned in the simple summary of novice web development (eight)-database HSQLDB instance .

1. Basic realization

Spring itself provides org.springframework.jdbc.datasource.DriverManagerDataSource, which can provide a simple database connection Connection object. Every time getConnection() is called, a new connection is generated. More suitable for unit testing or simple stand-alone applications.

• Because DriverManagerDataSource itself comes with springframework, you need to add dependencies in pom.xml

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-jdbc</artifactId>
            <version>5.2.5.RELEASE</version>
        </dependency>

• A simple summary of using Xiaobai novice web development (eight) -jdbc.properties in the database HSQLDB instance to configure the url, user, password, etc. of the database, as follows:

#hsqldb的相关配置
#最后会在apache/bin的目录下生成该数据库
jdbc.url = jdbc:hsqldb:file:db/hsqldb/xbook
jdbc.user = SA
jdbc.password =
jdbc.table = book
jdbc.driverClass=org.hsqldb.jdbcDriver

• Add an application-context-db.xml file under resources/config to configure the DriverManagerDataSource JavaBean

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context https://www.springframework.org/schema/context/spring-context.xsd">

    <!--增加数据库配置的配置文件-->
    <context:property-placeholder location="classpath:config/jdbc.properties"/>

    <!--简单的一个连接，每次都会新建-->
    <bean id="driverManagerDataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
        <property name="url" value="${jdbc.url}"/>
        <property name="username" value="${jdbc.user}"/>
        <property name="password" value="${jdbc.password}"/>
        <property name="driverClassName" value="${jdbc.driverClass}"/>
    </bean>

 </beans>

Because this DriverManagerDataSource creates a new Connection every time, it is not a database connection pool, so there is no need to configure parameters related to the database connection pool. Specifically, which important parameters need to be configured in the project to improve performance, etc., will be studied in the future.

•  Increase JdbcDataSource to get these DataSource instances

@Component
public class JdbcDataSource { 
    private DriverManagerDataSource driverManagerDataSource;

    public void setDriverManagerDataSource(DriverManagerDataSource driverManagerDataSource{
        this.driverManagerDataSource = driverManagerDataSource;
    }

    @PostConstruct
    public void init() {
        System.out.println("JdbcTemplate init  。。。。。。 ");
    }

    public DataSource createDriverManagerDataSource() {
        return driverManagerDataSource;
    }
}

• Then also increase the dependency between DriverManagerDataSource and JdbcDataSource in the application-context-db.xml file

    <!--处理数据库连接的bean-->
    <bean id="jdbcConfiguration" class="com.wj.hsqldb.datasource.JdbcDataSource">
        <property name="driverManagerDataSource" ref="driverManagerDataSource"/>
    </bean>

Remaining question: Why must the setDriverManagerDataSource() method be added to complete the configuration dependency  (I guess the probability is related to the binding of the dependency because of the reflection relationship, and the reflection process is to find the corresponding dependency through the setxxx method The attributes of the relationship).

• Finally, you need to read the application-context-db.xml in the web.xml file of the Spring project, and let the Spring IoC container load these JavaBeans.

    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>classpath:config/application-context*.xml</param-value>
    </context-param>

After these steps, you can directly use the driverManagerDataSource DataSource instance in the project. Then replace the previous com.wj.hsqldb.db.JdbcConfiguration with driverManagerDataSource.getConnection() through DriverManager.getConnection().

2. Database connection pool implementation

When a web application starts, the database connection pool will create a certain number of database connections and maintain no less than the maximum number of connections. When the web application is accessing the database, it will find an unused database connection from the database connection pool and mark the connection as busy. When there is no free connection, a new database connection will be created; when in use After the database connection is used up, the connection pool of the database will mark the connection as free, and then the web application will be available again when it is accessed.

From this process, we can see that the working mechanism of the thread pool is actually the same as that of Java, so it is not difficult to understand this place. So it can be seen that the advantages of DataSource are:

• 1) There is no need to frequently establish database connections and disconnections, saving resources and time;
• 2) There is no need to enter the user name and password frequently to connect to the database, saving memory and CPU overhead

Commonly used open source database connection pools in Java are as follows:

（1）DBCP

Database connection pool that relies on Jakarta commons-pool object pool mechanism. Can be used directly in web applications. DBCP is used in Tomcat. The corresponding implementation class is org.apache.commons.dbcp.BasicDataSource.

In the process of using the Spring project, the same as the DriverManagerDataSource, it is necessary to introduce dependencies and add configuration files in turn, and then use them directly in the project.

• 1) Add the dependency of BasicDataSource

        <!--DBCP数据源的配置-->
        <dependency>
            <groupId>commons-dbcp</groupId>
            <artifactId>commons-dbcp</artifactId>
            <version>1.4</version>
        </dependency>

• 2) Just add the JavaBean of BasicDataSource to the application-context-db.xml in (1) Basic Implementation , and add it to the JdbcDataSource dependency

    <!--配置DBCP数据源-->
    <bean id="dbcpDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
        <property name="username" value="${jdbc.user}"/>
        <property name="password" value="${jdbc.password}"/>
        <property name="url" value="${jdbc.url}"/>
        <property name="driverClassName" value="${jdbc.driverClass}"/>

        <property name="maxActive" value="6"/>
        <property name="maxIdle" value="3"/>
    </bean>
    <!--处理数据库连接的bean-->
    <bean id="jdbcConfiguration" class="com.wj.hsqldb.datasource.JdbcDataSource">
        <property name="dbcpDataSource" ref="dbcpDataSource"/>
        <property name="driverManagerDataSource" ref="driverManagerDataSource"/>
    </bean>

Because of the connection pool of the database created by BasicDataSource, it is necessary to configure some key parameters such as the minimum number of connections, the number of initial connections, etc., not to do research here, as the project will gradually learn about the configuration of these parameters on the performance of web applications accessing the database influences.

In addition, if you want to add this attribute of BasicDataSource to the attention of JdbcDataSource, you must add the dependency between the two in  (1) JdbcDataSource added in the basic implementation , otherwise there will be two instances of JdbcDataSource here, in another instance It is not possible to use attributes that are not defined.

• 3) Add a corresponding instance BasicDataSource in JdbcDataSource

@Component
public class JdbcDataSource {
    private BasicDataSource dbcpDataSource;
    private DriverManagerDataSource driverManagerDataSource;

    public void setDbcpDataSource(BasicDataSource dbcpDataSource) {
        this.dbcpDataSource = dbcpDataSource;
    }

    public void setDriverManagerDataSource(DriverManagerDataSource driverManagerDataSource{
        this.driverManagerDataSource = driverManagerDataSource;
    }

    @PostConstruct
    public void init() {
        System.out.println("JdbcTemplate init  。。。。。。 ");
    }

    public DataSource createBasicDataSource() {
        return dbcpDataSource;
    }

    public DataSource createDriverManagerDataSource() {
        return driverManagerDataSource;
    }
}

After these steps, you can directly use the DataSource instance of dbcpDataSource in the project. Then replace the previous com.wj.hsqldb.db.JdbcConfiguration with dbcpDataSource.getConnection() through DriverManager.getConnection(), and now create a Connection with a database connection pool. 

（2）C3P0

It has a richer configuration than DBCP, and the corresponding implementation class is com.mchange.v2.c3p0.ComboPooledDataSource. The specific usage is the same as above, please refer to the application-context-db.xml in the project and the relevant code in JdbcDataSource.

(3) HikariCP

The speed is higher than C3P0, the use of bytecode is more concise, more codes can be loaded into the cache, and a lock-free collection type can be realized. The database connection pool used by default in SpringBoot2.

（4）Druid

Alibaba product, Taobao and Alipay dedicated database connection pool, support all JDBC-compatible databases, including Oracle, MySql, Postgresql, etc., the corresponding implementation class com.alibaba.druid.pool.DruidDataSource. Replacing DBCP and C3P0, it provides an efficient, powerful, and scalable database connection pool.

It includes three parts:

• DruidDriver: Agent Driver, can monitor the access performance of the database;
• DruidDataSource: an efficient and manageable database connection pool
• SQLParse: Get the SQL execution log.

The specific usage is the same as above, please refer to the application-context-db.xml in the project and the related code in JdbcDataSource.

Then after configuring the above four data sources, the codes of the last modified files are as follows:

• 1) The dependencies added by pom.xml are:

  <!--jdbc接口 Spring对JDBC的封装-->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-jdbc</artifactId>
            <version>5.2.5.RELEASE</version>
        </dependency>
        <!-- === 配置数据源  ==== -->
        <!--c3p0数据源-->
        <dependency>
            <groupId>c3p0</groupId>
            <artifactId>c3p0</artifactId>
            <version>0.9.1.2</version>
        </dependency>
        <!--DBCP数据源的配置-->
        <dependency>
            <groupId>commons-dbcp</groupId>
            <artifactId>commons-dbcp</artifactId>
            <version>1.4</version>
        </dependency>
        <!--druid-->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>druid</artifactId>
            <version>1.2.5</version>
        </dependency>

•  2）application-context-db.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context https://www.springframework.org/schema/context/spring-context.xsd">

    <!--增加数据库配置的配置文件-->
    <context:property-placeholder location="classpath:config/jdbc.properties"/>

    <!--简单的一个连接，每次都会新建-->
    <bean id="driverManagerDataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
        <property name="url" value="${jdbc.url}"/>
        <property name="username" value="${jdbc.user}"/>
        <property name="password" value="${jdbc.password}"/>
        <property name="driverClassName" value="${jdbc.driverClass}"/>
    </bean>

    <!--数据库连接池-->

    <!--配置C3P0数据源-->
    <bean id="c3p0DataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">
        <property name="user" value="${jdbc.user}"/>
        <property name="password" value="${jdbc.password}"/>
        <property name="jdbcUrl" value="${jdbc.url}"/>
        <property name="driverClass" value="${jdbc.driverClass}"/>

        <property name="initialPoolSize" value="3"/>
        <property name="maxPoolSize" value="6"/>
    </bean>
    <!--配置DBCP数据源-->
    <bean id="dbcpDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
        <property name="username" value="${jdbc.user}"/>
        <property name="password" value="${jdbc.password}"/>
        <property name="url" value="${jdbc.url}"/>
        <property name="driverClassName" value="${jdbc.driverClass}"/>

        <property name="maxActive" value="6"/>
        <property name="maxIdle" value="3"/>
    </bean>
    <!--阿里的-->
    <bean id="druidDataSource" class="com.alibaba.druid.pool.DruidDataSource" destroy-method="close">
        <property name="username" value="${jdbc.user}"/>
        <property name="password" value="${jdbc.password}"/>
        <property name="url" value="${jdbc.url}"/>
        <property name="driverClassName" value="${jdbc.driverClass}"/>
    </bean>

    <!--处理数据库连接的bean-->
    <bean id="jdbcConfiguration" class="com.wj.hsqldb.datasource.JdbcDataSource">
        <property name="c3p0DataSource" ref="c3p0DataSource"/>
        <property name="dbcpDataSource" ref="dbcpDataSource"/>
        <property name="driverManagerDataSource" ref="driverManagerDataSource"/>
        <property name="druidDataSource" ref="druidDataSource"/>
    </bean>

</beans>

• 3）JdbcDataSource

@Component
public class JdbcDataSource {
    private ComboPooledDataSource c3p0DataSource;
    private BasicDataSource dbcpDataSource;
    private DriverManagerDataSource driverManagerDataSource;
    private DruidDataSource druidDataSource;

    public void setC3p0DataSource(ComboPooledDataSource c3p0DataSource) {
        this.c3p0DataSource = c3p0DataSource;
    }

    public void setDbcpDataSource(BasicDataSource dbcpDataSource) {
        this.dbcpDataSource = dbcpDataSource;
    }

    public void setDriverManagerDataSource(DriverManagerDataSource driverManagerDataSource{
        this.driverManagerDataSource = driverManagerDataSource;
    }

    public void setDruidDataSource(DruidDataSource druidDataSource) {
        this.druidDataSource = druidDataSource;
    }

    @PostConstruct
    public void init() {
        System.out.println("JdbcTemplate init  。。。。。。 ");

    }

    public DataSource createComboPoolDataSource() {
        return c3p0DataSource;
    }

    public DataSource createBasicDataSource() {
        return dbcpDataSource;
    }

    public DataSource createDriverManagerDataSource() {
        return driverManagerDataSource;
    }

    public DataSource createDruidDataSource() {
        return druidDataSource;
    }
}

The relevant code has been uploaded to github: https://github.com/wenjing-bonnie/sql-web.git The corresponding tag is the relevant code under example11 (because the project is always updated, so this code is marked by tagging ) .

3. Distributed implementation

In some complex web application development, an application may involve connecting to multiple databases, that is, multiple data sources. List two commonly used common scenarios:

• 1) Read-write separated database: The read database is only responsible for various query operations, and the write database is responsible for additions, deletions and modifications;
• 2) Complex business: A business process may need to read or modify data in two databases at the same time

This kind of multi-data source application is a typical distributed scenario. In the Spring project, spring-jdbc provides org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource, which can contain multiple DataSources, which can dynamically access the current data source during runtime according to the rules defined by the developer.

How to use this AbstractRoutingDataSource in the project, I think I will look at it according to the project. Now I know that this can load multiple data sources, but how to use it in the project needs to be followed by the project to study.

Leftover question: How to use multiple data sources in the project? ?

Three summary 

Summary of problems encountered in the process of writing code examples:

1.Mixed use of annotations in Spring and xml configuration JavaBean

In web application development, it is difficult to avoid that some JavaBeans will be instantiated through annotations, and some JavaBeans will be configured through xml. Then it will appear in the class annotated by @Component. How to refer to an instance configured in xml? ?

@AutoWrited is automatically injected according to type, then @Resource is automatically injected according to name. There are two important attributes: name and type: if these two attributes are not specified, it will search by name by default

In @Resource, the name is to match <bean name=""/> in the configured xml file, not <bean id=""/>. If no matching bean is found, it will fall back to a primitive type. Search; if you search according to the type attribute, you will find the only bean with a matching type for injection, if you cannot find or more than one, an exception will be thrown; if both are matched, the only bean will be found for matching , Throw an exception if not found.

@Resource is equivalent to @Autowired, except that @Autowired is automatically injected according to byType, and @Resource is automatically injected according to byName by default. Both @Autowired and @Resource can be used to assemble beans. Both can be written on fields or setter methods.

For example, if you want to use the JdbcDataSource configured in xml in BookManagerJdbcService.java, you have the following code

@Configuration
public class BookManagerJdbcService {
    @Resource(type = JdbcDataSource.class, name = "jdbcDataSource")
    private JdbcDataSource jdbcDataSource;
}

The code in application-context-db.xml is as follows:

    <!--处理数据库连接的bean-->
    <bean id="jdbcDataSource" class="com.wj.hsqldb.datasource.JdbcDataSource" name="jdbcDataSource">
        <property name="c3p0DataSource" ref="c3p0DataSource"/>
        <property name="dbcpDataSource" ref="dbcpDataSource"/>
        <property name="driverManagerDataSource" ref="driverManagerDataSource"/>
        <property name="druidDataSource" ref="druidDataSource"/>
    </bean>

 In this way, you can directly use the jdbcDataSource instance in BookManagerJdbcService.java.

2. The web application does not directly access the database, but accesses the database driver through the JDBC interface, thereby accessing the database;

3. The database driver is generally provided by the database manufacturer, which not only includes the logic of accessing the database, but also includes the underlying network communication;

4. DataSource is a data source, which provides a database connection for connecting to the database;

5. DataSource is divided into three types: one is to provide only one Connection at a time; the second is a database connection pool (similar to the concept of thread pool); the third is a distributed data source;

6. Usually a DataSource needs to configure the basic configuration of the database url, user name, password and driverClass, as well as some performance configurations and key configurations;

Of course, there are some remaining issues that need to be gradually understood later

1. Why must the setDriverManagerDataSource() method be added to complete the configuration dependency? 

2. How to use multiple data sources in the project? ?

The next article is to summarize the simple summary of the novice web development of Xiaobai (12)-related optimization of database connection (transaction management)