SpringBoot——Enable Druid monitoring statistics function (SQL monitoring, slow SQL records, Spring monitoring, advertising removal)

reference:

github#Druid source code

github#druid-spring-boot-startersource code

Druid Wiki

springboot#data-properties

reference blog

SpringBoot+Druid+mybatis+log4j2+mysql8

---

1 Basic concepts

Druid is the best database connection pool in the Java language.

Although HikariCP is slightly faster, Druid can provide powerful monitoring and expansion functions , and it is also an open source project of Alibaba.

Druid is a database connection pool developed by Alibaba known as monitoring. It surpasses other database connection pools in terms of function, performance, and scalability, including DBCP, C3P0, BoneCP, Proxool, JBoss DataSource, etc., killing everything in seconds.

Druid can monitor the DB pool connection and SQL execution very well, and it is a DB connection pool for monitoring.

Spring Boot's default data source HikariDataSource and JdbcTemplate have introduced Spring Boot 2.x to use the Hikari data source by default . It can be said that Hikari and Driud are the best data sources on the Java Web.

And Druid has deployed more than 600 applications in Alibaba, after several years of severe tests of large-scale deployment in the production environment!

• stat: Druid provides one built-in StatFilterfor statistical monitoring information.
• wall: Druid's defense against SQL injection attacks WallFilteris through Druid's SQL Parser analysis. The SQL Parser provided by Druid can intercept SQL at the JDBC layer for corresponding processing, such as sub-database sub-table, auditing, etc.
• log4j2: This is the logging function, which can print sql statements to log4j2 for troubleshooting.

---

2 Add dependencies

pom.xml

<!-- 阿里巴巴的druid数据源 -->
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>druid-spring-boot-starter</artifactId>
    <version>1.1.23</version>
</dependency>
<!-- mysql8 驱动-->
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <scope>runtime</scope>
</dependency>
<!--使用 log4j2 记录日志-->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>

<!--
mybatis，引入了 SpringBoot的 JDBC 模块，
所以，默认是使用 hikari 作为数据源
-->
<dependency>
    <groupId>org.mybatis.spring.boot</groupId>
    <artifactId>mybatis-spring-boot-starter</artifactId>
    <version>2.1.3</version>
    <exclusions>
        <!-- 排除默认的 HikariCP 数据源 -->
        <exclusion>
            <groupId>com.zaxxer</groupId>
            <artifactId>HikariCP</artifactId>
        </exclusion>
    </exclusions>
</dependency>

3 Configure related properties

• Configure the Druid data source (connection pool) : Just like the previous c3p0 and dbcp data sources can set the data source connection initialization size, maximum number of connections, waiting time, minimum number of connections, etc., the Druid data source can be set in the same way;
• Configure Druid web monitoring filter ( WebStatFilter) : The function of this filter is to count all database information in web application requests, such as issued sql statements, sql execution time, number of requests, requested url address, and seesion monitoring, database table number of visits, etc.
• Configure Druid background management Servlet ( StatViewServlet) : The Druid data source has the function of monitoring, and provides a web interface for users to view, similar to when installing a router, they also provide a default web page; you need to set the properties of the Druid background management page , such as login account, password, etc.;

Notice:

The name of the Druid Spring Boot Starter configuration property fully follows Druid, and the Druid database connection pool and monitoring can be configured through the Spring Boot configuration file. If there is no configuration, the default value will be used.

application.yml

########## 配置数据源 （Druid）##########
spring:
  datasource:
    ########## JDBC 基本配置 ##########
    username: xxx
    password: xxx
    driver-class-name: com.mysql.cj.jdbc.Driver   # mysql8 的连接驱动
    url: jdbc:mysql://127.0.0.1:3306/test?serverTimezone=Asia/Shanghai
    platform: mysql                               # 数据库类型
    type: com.alibaba.druid.pool.DruidDataSource  # 指定数据源类型
    ########## 连接池 配置 ##########
    druid:
      # 配置初始化大小、最小、最大
      initial-size: 5
      minIdle: 10
      max-active: 20
      # 配置获取连接等待超时的时间(单位：毫秒)
      max-wait: 60000
      # 配置间隔多久才进行一次检测，检测需要关闭的空闲连接，单位是毫秒
      time-between-eviction-runs-millis: 2000
      # 配置一个连接在池中最小生存的时间，单位是毫秒
      min-evictable-idle-time-millis: 600000
      max-evictable-idle-time-millis: 900000
      # 用来测试连接是否可用的SQL语句,默认值每种数据库都不相同,这是mysql
      validationQuery: select 1
      # 应用向连接池申请连接，并且testOnBorrow为false时，连接池将会判断连接是否处于空闲状态，如果是，则验证这条连接是否可用
      testWhileIdle: true
      # 如果为true，默认是false，应用向连接池申请连接时，连接池会判断这条连接是否是可用的
      testOnBorrow: false
      # 如果为true（默认false），当应用使用完连接，连接池回收连接的时候会判断该连接是否还可用
      testOnReturn: false
      # 是否缓存preparedStatement，也就是PSCache。PSCache对支持游标的数据库性能提升巨大，比如说oracle
      poolPreparedStatements: true
      # 要启用PSCache，必须配置大于0，当大于0时， poolPreparedStatements自动触发修改为true，
      # 在Druid中，不会存在Oracle下PSCache占用内存过多的问题，
      # 可以把这个数值配置大一些，比如说100
      maxOpenPreparedStatements: 20
      # 连接池中的minIdle数量以内的连接，空闲时间超过minEvictableIdleTimeMillis，则会执行keepAlive操作
      keepAlive: true
      # Spring 监控，利用aop 对指定接口的执行时间，jdbc数进行记录
      aop-patterns: "com.springboot.template.dao.*"
      ########### 启用内置过滤器（第一个 stat必须，否则监控不到SQL）##########
      filters: stat,wall,log4j2
      # 自己配置监控统计拦截的filter
      filter:
        # 开启druiddatasource的状态监控
        stat:
          enabled: true
          db-type: mysql
          # 开启慢sql监控，超过2s 就认为是慢sql，记录到日志中
          log-slow-sql: true
          slow-sql-millis: 2000
        # 日志监控，使用slf4j 进行日志输出
        slf4j:
          enabled: true
          statement-log-error-enabled: true
          statement-create-after-log-enabled: false
          statement-close-after-log-enabled: false
          result-set-open-after-log-enabled: false
          result-set-close-after-log-enabled: false
      ########## 配置WebStatFilter，用于采集web关联监控的数据 ##########
      web-stat-filter:
        enabled: true                   # 启动 StatFilter
        url-pattern: /*                 # 过滤所有url
        exclusions: "*.js,*.gif,*.jpg,*.png,*.css,*.ico,/druid/*" # 排除一些不必要的url
        session-stat-enable: true       # 开启session统计功能
        session-stat-max-count: 1000    # session的最大个数,默认100
      ########## 配置StatViewServlet（监控页面），用于展示Druid的统计信息 ##########
      stat-view-servlet:
        enabled: true                   # 启用StatViewServlet
        url-pattern: /druid/*           # 访问内置监控页面的路径，内置监控页面的首页是/druid/index.html
        reset-enable: false              # 不允许清空统计数据,重新计算
        login-username: root            # 配置监控页面访问密码
        login-password: 123
        allow: 127.0.0.1           # 允许访问的地址，如果allow没有配置或者为空，则允许所有访问
        deny:                                        # 拒绝访问的地址，deny优先于allow，如果在deny列表中，就算在allow列表中，也会被拒绝

The parameters of the above configuration files com.alibaba.druid.spring.boot.autoconfigure.properties.DruidStatPropertiescan org.springframework.boot.autoconfigure.jdbc.DataSourcePropertiesbe found in and;

3.1 How to configure Filter

The corresponding built-in Filter can be enabled spring.datasource.druid.filters=stat,wall,log4j ...by way, but these Filters are default configurations. If the default configuration cannot meet the requirements, you can abandon this method and configure Filter through the configuration file. The following is an example.

# 配置StatFilter 
spring.datasource.druid.filter.stat.enabled=true
spring.datasource.druid.filter.stat.db-type=h2
spring.datasource.druid.filter.stat.log-slow-sql=true
spring.datasource.druid.filter.stat.slow-sql-millis=2000

# 配置WallFilter 
spring.datasource.druid.filter.wall.enabled=true
spring.datasource.druid.filter.wall.db-type=h2
spring.datasource.druid.filter.wall.config.delete-allow=false
spring.datasource.druid.filter.wall.config.drop-table-allow=false

Currently, configuration support is provided for the following Filters, which are configured according to (spring.datasource.druid.filter.*).

• StatFilter
• WallFilter
• ConfigFilter
• EncodingConvertFilter
• Slf4jLogFilter
• Log4jFilter
• Log4j2Filter
• CommonsLogFilter

If you don't want to use the built-in Filters, you need to set the corresponding Filter's enabled to true to make the custom Filter configuration take effect. Druid Spring Boot Starter disables StatFilter by default, and you can set its enabled to true to enable it.

3 Monitoring page

(1) After starting the project, visit /druid/login.htmlto the login page , enter the user name and password to log in

(2) The data source page is the basic information of the current DataSource configuration, the Filter configured above can be found in it, if the Filter is not configured (some information will not be available) Statistics, such as "SQL monitoring", will not be able to obtain JDBC-related SQL execution information)

(3) SQL monitoring page , statistics of the execution of all SQL statements

(4) URL monitoring page , statistics of access and execution of all Controller interfaces

(5) Spring monitoring page, using aop to record the execution time and jdbc number of the specified interface

(6) SQL firewall page

Druid provides access to the black and white lists, and you can clearly see the sql protection situation.

(7) Session monitoring page

You can see the current session status, detailed parameters such as creation time, last active time, number of requests, and request time.

(8) JSON API page

Access Druid's monitoring interface in the form of api, and the api interface returns data in the form of Json.

---

4 sql monitoring

Configure the Druid web monitoring filter ( WebStatFilter), which is used to count all database information in web application requests, such as issued sql statements, sql execution time, number of requests, requested url addresses, and seesion monitoring and access to database tables times and so on.

spring:
  datasource:
    druid:
      ########## 配置WebStatFilter，用于采集web关联监控的数据 ##########
      web-stat-filter:
        enabled: true                   # 启动 StatFilter
        url-pattern: /*                 # 过滤所有url
        exclusions: "*.js,*.gif,*.jpg,*.png,*.css,*.ico,/druid/*" # 排除一些不必要的url
        session-stat-enable: true       # 开启session统计功能
        session-stat-max-count: 1000    # session的最大个数,默认100

5 slow sql records

Sometimes, some SQL in the system executes very slowly, and we hope to use logs to record them, so we can enable Druid's slow SQL recording function

spring:
  datasource:
    druid:
      filter:
        stat:
          enabled: true         # 开启DruidDataSource状态监控
          db-type: mysql        # 数据库的类型
          log-slow-sql: true    # 开启慢SQL记录功能
          slow-sql-millis: 2000 # 默认3000毫秒，这里超过2s，就是慢，记录到日志

After startup, if you encounter slow SQL, it will be output to the log

6 spring monitoring

After access, spring monitoring has no data by default;

This needs to import the AOP Starter of SprngBoot

<!--SpringBoot 的aop 模块-->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

Need to be configured in application.yml:

Spring monitoring AOP entry point, such as com.springboot.template.dao.*, configure multiple English comma-separated

 spring.datasource.druid.aop-patterns="com.springboot.template.dao.*"

7 Go to Ad (advertisement)

When visiting the monitoring page, you may see Alibaba's advertisement at the bottom of the page (footer)

Reason: In the imported druid jar package common.js(there is a piece of js code in it to add advertisements to the footer of the page)

If you want to remove it, there are two ways:

(1) Manually comment this code directly

If you are using Maven, go directly to the local warehouse to find the jar package

Code to comment:

// this.buildFooter();

The location of common.js:

com/alibaba/druid/1.1.23/druid-1.1.23.jar!/support/http/resources/js/common.js

(2) Filter using a filter

Register a filter, filter common.js requests, and use regular expressions to replace relevant advertising content

@Configuration
@ConditionalOnWebApplication
@AutoConfigureAfter(DruidDataSourceAutoConfigure.class)
@ConditionalOnProperty(name = "spring.datasource.druid.stat-view-servlet.enabled",
havingValue = "true", matchIfMissing = true)
public class RemoveDruidAdConfig {
    
    

    /**
    * 方法名: removeDruidAdFilterRegistrationBean
    * 方法描述 除去页面底部的广告
    * @param properties com.alibaba.druid.spring.boot.autoconfigure.properties.DruidStatProperties
    * @return org.springframework.boot.web.servlet.FilterRegistrationBean
    */
    @Bean
    public FilterRegistrationBean removeDruidAdFilterRegistrationBean(DruidStatProperties properties) {
    
    

        // 获取web监控页面的参数
        DruidStatProperties.StatViewServlet config = properties.getStatViewServlet();
        // 提取common.js的配置路径
        String pattern = config.getUrlPattern() != null ? config.getUrlPattern() : "/druid/*";
        String commonJsPattern = pattern.replaceAll("\\*", "js/common.js");

        final String filePath = "support/http/resources/js/common.js";

        //创建filter进行过滤
        Filter filter = new Filter() {
    
    
            @Override
            public void init(FilterConfig filterConfig) throws ServletException {
    
    }

            @Override
            public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
    
    
                chain.doFilter(request, response);
                // 重置缓冲区，响应头不会被重置
                response.resetBuffer();
                // 获取common.js
                String text = Utils.readFromResource(filePath);
                // 正则替换banner, 除去底部的广告信息
                text = text.replaceAll("<a.*?banner\"></a><br/>", "");
                text = text.replaceAll("powered.*?shrek.wang</a>", "");
                response.getWriter().write(text);
            }

            @Override
            public void destroy() {
    
    }
        };

        FilterRegistrationBean registrationBean = new FilterRegistrationBean();
        registrationBean.setFilter(filter);
        registrationBean.addUrlPatterns(commonJsPattern);
        return registrationBean;
    }
}

Both methods are available, it is recommended to use the first method to solve the problem from the root cause

8 Obtain Druid monitoring data

The monitoring data of Druid canDruidStatManagerFacade be obtained through after opening StatFilter;

DruidStatManagerFacade#getDataSourceStatDataListThis method can obtain the monitoring data of all data sources,

In addition, some other methods DruidStatManagerFacadeare provided, which can be selected and used according to needs.

@RestController
@RequestMapping(value = "/druid")
public class DruidStatController {
    
    

    @GetMapping("/stat")
    public Object druidStat(){
    
    
        // 获取数据源的监控数据
        return DruidStatManagerFacade.getInstance().getDataSourceStatDataList();
    }
}