MySQL batch Insert, HikariCP, thread pool parameter optimization test (continued)

The first part of MySQL batch insert, HikariCP, thread pool parameter optimization test describes the exploration test of optimization parameters during MySQL batch insert. When I studied the optimization parameters in depth, I found that MySQL did not really enable the rewriteBatchedStatements parameter in the previous article. Turning on this parameter can greatly improve the write performance.

rewriteBatchedStatements=true

MySQL turns off batch processing by default, and it is turned on by this parameter. This parameter can rewrite the SQL statement submitted to the database. For details, see: http://www.cnblogs.com/chenjianjx/archive/2012/08/14/2637914. html

Configuration

    static{
    
    

        HikariConfig config = new HikariConfig();
      config.setJdbcUrl("jdbc:mysql://192.168.1.1:3306/test");
      config.setUsername("xxxx");
      config.setPassword("xxxx");
      config.addDataSourceProperty("cachePrepStmts", "true");
      config.addDataSourceProperty("prepStmtCacheSize", "250");
      config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048");

       config.addDataSourceProperty("rewriteBatchedStatements", "true");
      
      config.setMaximumPoolSize(30);
      config.setConnectionTimeout(6000);
      config.setValidationTimeout(3000);
      
      try {
    
    
        ds = new HikariDataSource(config);
        Properties prop = ds.getDataSourceProperties();
        System.out.println("rewriteBatchedStatements = " + prop.getProperty("rewriteBatchedStatements"));
      } catch (Exception e) {
    
    
    	  ds = null;
      }
    }

Test Results

HikariCP connection pool 30 connections, 30 worker threads

--------C组测试----------
单条插入50000条记录，耗时：86.3秒, 平均:579.38 TPS！ 连接耗时:0.693
批量插入25批 * 2000条/批  (共50000条)，耗时:2.13秒, 平均:23441.16 TPS！ 连接耗时:0.0
批量插入50批 * 1000条/批  (共50000条)，耗时:2.51秒, 平均:19928.26 TPS！ 连接耗时:0.092
批量插入100批 * 500条/批  (共50000条)，耗时:2.77秒, 平均:18024.51 TPS！ 连接耗时:0.625
批量插入200批 * 250条/批  (共50000条)，耗时:3.06秒, 平均:16350.56 TPS！ 连接耗时:0.0
批量插入400批 * 125条/批  (共50000条)，耗时:3.61秒, 平均:13861.93 TPS！ 连接耗时:0.032
批量插入1000批 * 50条/批  (共50000条)，耗时:5.46秒, 平均:9150.80 TPS！ 连接耗时:0.0
批量插入2500批 * 20条/批  (共50000条)，耗时:9.29秒, 平均:5380.97 TPS！ 连接耗时:0.006
C组测试过程结束，全部测试耗时：117.158秒！
--------D组测试----------
单条插入50000条记录，耗时：75.39秒, 平均:663.19 TPS！ 连接耗时:0.092
批量插入25批 * 2000条/批  (共50000条)，耗时:1.56秒, 平均:32030.75 TPS！ 连接耗时:0.183
批量插入50批 * 1000条/批  (共50000条)，耗时:1.69秒, 平均:29655.99 TPS！ 连接耗时:0.107
批量插入100批 * 500条/批  (共50000条)，耗时:1.87秒, 平均:26795.28 TPS！ 连接耗时:0.02
批量插入200批 * 250条/批  (共50000条)，耗时:2.73秒, 平均:18301.61 TPS！ 连接耗时:0.001
批量插入400批 * 125条/批  (共50000条)，耗时:2.86秒, 平均:17458.1 TPS！ 连接耗时:0.002
批量插入1000批 * 50条/批  (共50000条)，耗时:4.22秒, 平均:11842.73 TPS！ 连接耗时:0.002
批量插入2500批 * 20条/批  (共50000条)，耗时:6.80秒, 平均:7349.7 TPS！ 连接耗时:0.006
D组测试过程结束，全部测试耗时：98.997秒！

summary

a) Compared with the test results in the previous MySQL batch Insert, HikariCP, and thread pool parameter optimization tests , enabling the rewriteBatchedStatements parameter, the performance is about 5 times higher!
Batch update… where operation has the highest throughput compared to the test result of only insert into
Batch insert into… on duplicate update operation
- If the on duplicate hit rate is zero, compared to the test result of only insert into, the throughput has almost no impact
- If the on duplicate hit rate is 1/5, the throughput will decrease significantly compared to the test result with only insert into

The following is the batch insert into, update, insert into… on duplicate update (zero update, 20% update) throughput test comparison:
Insert picture description here