Bulk insert for data (jdbc \ mybatis)
1. implementation - JDBC:
- Common for loop inserted
1 private String url = "jdbc:mysql://localhost:3306/test01";
2 private String user = "root";
3 private String password = "123456";
4 @Test
5 public void Test(){
6 Connection conn = null;
7 PreparedStatement pstm =null;
8 ResultSet rt = null;
9 try {
10 Class.forName("com.mysql.jdbc.Driver");
11 conn = DriverManager.getConnection(url, user, password);
12 String sql = "INSERT INTO userinfo(uid,uname,uphone,uaddress) VALUES(?,CONCAT('姓名',?),?,?)";
13 pstm = conn.prepareStatement(sql);
14 Long startTime = System.currentTimeMillis();
15 Random rand = new Random();
16 int a,b,c,d;
17 for (int i = 1; i <= 1000; i++) {
18 pstm.setInt(1, i);
19 pstm.setInt(2, i);
20 a = rand.nextInt(10);
21 b = rand.nextInt(10);
22 c = rand.nextInt(10);
23 d = rand.nextInt(10);
24 pstm.setString(3, "188"+a+"88"+b+c+"66"+d);
25 pstm.setString(4, "xxxxxxxxxx_"+"188"+a+"88"+b+c+"66"+d);27 pstm.executeUpdate();
28 }
29 Long endTime = System.currentTimeMillis();
30 System.out.println("OK,用时：" + (endTime - startTime));
31 } catch (Exception e) {
32 e.printStackTrace();
33 throw new RuntimeException(e);
34 }finally{
35 if(pstm!=null){
36 try {
37 pstm.close();
38 } catch (SQLException e) {
39 e.printStackTrace();
40 throw new RuntimeException(e);
41 }
42 }
43 if(conn!=null){
44 try {
45 conn.close();
46 } catch (SQLException e) {
47 e.printStackTrace();
48 throw new RuntimeException(e);
49 }
50 }
51 }
52 }
- Use transaction inserted
first commit command mode set to false, that the author conn.setAutoCommit (false) manual; and finally commit the transaction conn.commit after all the command finishes ();
private String url = "jdbc:mysql://localhost:3306/test01";
2 private String user = "root";
3 private String password = "123456";
4 @Test
5 public void Test(){
6 Connection conn = null;
7 PreparedStatement pstm =null;
8 ResultSet rt = null;
9 try {
10 Class.forName("com.mysql.jdbc.Driver");
11 conn = DriverManager.getConnection(url, user, password);
12 String sql = "INSERT INTO userinfo(uid,uname,uphone,uaddress) VALUES(?,CONCAT('姓名',?),?,?)";
13 pstm = conn.prepareStatement(sql);
14 conn.setAutoCommit(false);
15 Long startTime = System.currentTimeMillis();
16 Random rand = new Random();
17 int a,b,c,d;
18 for (int i = 1; i <= 100000; i++) {
19 pstm.setInt(1, i);
20 pstm.setInt(2, i);
21 a = rand.nextInt(10);
22 b = rand.nextInt(10);
23 c = rand.nextInt(10);
24 d = rand.nextInt(10);
25 pstm.setString(3, "188"+a+"88"+b+c+"66"+d);
26 pstm.setString(4, "xxxxxxxxxx_"+"188"+a+"88"+b+c+"66"+d);
27 pstm.executeUpdate();
28 }
29 conn.commit();
30 Long endTime = System.currentTimeMillis();
31 System.out.println("OK,用时：" + (endTime - startTime));
32 } catch (Exception e) {
33 e.printStackTrace();
34 throw new RuntimeException(e);
35 }finally{
36 if(pstm!=null){
37 try {
38 pstm.close();
39 } catch (SQLException e) {
40 e.printStackTrace();
41 throw new RuntimeException(e);
42 }
43 }
44 if(conn!=null){
45 try {
46 conn.close();
47 } catch (SQLException e) {
48 e.printStackTrace();
49 throw new RuntimeException(e);
50 }
51 }
52 }
53 }
- Batch processing
First, url JDBC connection parameter is set to be added rewriteBatchedStatements premise is true batch operation, followed by mysql driver package is to check the time is 5.1.13 or later (below this version does not support)
private String url = "jdbc:mysql://localhost:3306/test01?rewriteBatchedStatements=true";
2 private String user = "root";
3 private String password = "123456";
4 @Test
5 public void Test(){
6 Connection conn = null;
7 PreparedStatement pstm =null;
8 ResultSet rt = null;
9 try {
10 Class.forName("com.mysql.jdbc.Driver");
11 conn = DriverManager.getConnection(url, user, password);
12 String sql = "INSERT INTO userinfo(uid,uname,uphone,uaddress) VALUES(?,CONCAT('姓名',?),?,?)";
13 pstm = conn.prepareStatement(sql);
14 Long startTime = System.currentTimeMillis();
15 Random rand = new Random();
16 int a,b,c,d;
17 for (int i = 1; i <= 100000; i++) {
18 pstm.setInt(1, i);
19 pstm.setInt(2, i);
20 a = rand.nextInt(10);
21 b = rand.nextInt(10);
22 c = rand.nextInt(10);
23 d = rand.nextInt(10);
24 pstm.setString(3, "188"+a+"88"+b+c+"66"+d);
25 pstm.setString(4, "xxxxxxxxxx_"+"188"+a+"88"+b+c+"66"+d);
26 pstm.addBatch();
27 }
28 pstm.executeBatch();
29 Long endTime = System.currentTimeMillis();
30 System.out.println("OK,用时：" + (endTime - startTime));
31 } catch (Exception e) {
32 e.printStackTrace();
33 throw new RuntimeException(e);
34 }finally{
35 if(pstm!=null){
36 try {
37 pstm.close();
38 } catch (SQLException e) {
39 e.printStackTrace();
40 throw new RuntimeException(e);
41 }
42 }
43 if(conn!=null){
44 try {
45 conn.close();
46 } catch (SQLException e) {
47 e.printStackTrace();
48 throw new RuntimeException(e);
49 }
50 }
51 }
52 }
- Transactional operations + batch processing mode
private String url = "jdbc:mysql://localhost:3306/test01?rewriteBatchedStatements=true";
2 private String user = "root";
3 private String password = "123456";
4 @Test
5 public void Test(){
6 Connection conn = null;
7 PreparedStatement pstm =null;
8 ResultSet rt = null;
9 try {
10 Class.forName("com.mysql.jdbc.Driver");
11 conn = DriverManager.getConnection(url, user, password);
12 String sql = "INSERT INTO userinfo(uid,uname,uphone,uaddress) VALUES(?,CONCAT('姓名',?),?,?)";
13 pstm = conn.prepareStatement(sql);
14 conn.setAutoCommit(false);
15 Long startTime = System.currentTimeMillis();
16 Random rand = new Random();
17 int a,b,c,d;
18 for (int i = 1; i <= 100000; i++) {
19 pstm.setInt(1, i);
20 pstm.setInt(2, i);
21 a = rand.nextInt(10);
22 b = rand.nextInt(10);
23 c = rand.nextInt(10);
24 d = rand.nextInt(10);
25 pstm.setString(3, "188"+a+"88"+b+c+"66"+d);
26 pstm.setString(4, "xxxxxxxxxx_"+"188"+a+"88"+b+c+"66"+d);
27 pstm.addBatch();
28 }
29 pstm.executeBatch();
30 conn.commit();
31 Long endTime = System.currentTimeMillis();
32 System.out.println("OK,用时：" + (endTime - startTime));
33 } catch (Exception e) {
34 e.printStackTrace();
35 throw new RuntimeException(e);
36 }finally{
37 if(pstm!=null){
38 try {
39 pstm.close();
40 } catch (SQLException e) {
41 e.printStackTrace();
42 throw new RuntimeException(e);
43 }
44 }
45 if(conn!=null){
46 try {
47 conn.close();
48 } catch (SQLException e) {
49 e.printStackTrace();
50 throw new RuntimeException(e);
51 }
52 }
53 }
54 }
2 implementation - mybatis
the foreach loop in mybatis
<insert id="batchInsert" parameterType="java.util.List">
insert into USER (id, name) values
<foreach collection="list" item="model" index="index" separator=",">
(#{model.id}, #{model.name})
</foreach>
</insert>
The principle is the traditional single insert statement
INSERT INTO `table1` (`field1`, `field2`) VALUES ("data1", "data2");
INSERT INTO `table1` (`field1`, `field2`) VALUES ("data1", "data2");
INSERT INTO `table1` (`field1`, `field2`) VALUES ("data1", "data2");
INSERT INTO `table1` (`field1`, `field2`) VALUES ("data1", "data2");
INSERT INTO `table1` (`field1`, `field2`) VALUES ("data1", "data2");
Convert:
INSERT INTO `table1` (`field1`, `field2`) VALUES ("data1", "data2"),
("data1", "data2"),
("data1", "data2"),
("data1", "data2"),
("data1", "data2");
Ideally, this data may be sent a number of new rows in a single one-time connection, and the delay and all index updates the last to perform consistency checks.
Foreach first glance this is no problem, but after practice, the project found that when the number of columns in the table more (20+), and the number of rows of more disposable insert (5000+), the entire insertion takes very long reach 14 minutes, which can not tolerate.
mybatis default actuator type is Simple, creates a prepared statement for each new statement, which is to create a PreparedStatement object. In our project, we will continue to use bulk insert this method, but because MyBatis to statements contained in the cache can not be adopted, then every method invocation, will re-parse the sql statement.
So, if you must use foreach batch manner, then the insertion may be considered to reduce the number of values of an insert statement, preferably to a value above the bottommost curve, so that the fastest. According to the general experience, 20 to 50-time interpolation is more appropriate number of rows, the time consumed can accept.
Focus here. Mentioned above it is that if you must insert a manner, a manner can improve performance. In fact, MyBatis document written bulk insert when it is recommended to use another method. (You can see http://www.mybatis.org/mybatis-dynamic-sql/docs/insert.html in Batch Insert Support title in the content)
SqlSession session = sqlSessionFactory.openSession(ExecutorType.BATCH);
try {
SimpleTableMapper mapper = session.getMapper(SimpleTableMapper.class);
List<SimpleTableRecord> records = getRecordsToInsert(); // not shown
BatchInsert<SimpleTableRecord> batchInsert = insert(records)
.into(simpleTable)
.map(id).toProperty("id")
.map(firstName).toProperty("firstName")
.map(lastName).toProperty("lastName")
.map(birthDate).toProperty("birthDate")
.map(employed).toProperty("employed")
.map(occupation).toProperty("occupation")
.build()
.render(RenderingStrategy.MYBATIS3);
batchInsert.insertStatements().stream().forEach(mapper::insert);
session.commit();
} finally {
session.close();
}
That basic idea is to set MyBatis session of the executor type Batch, and then execute insert statement many times. The following statement is similar to JDBC same.
Connection connection = DriverManager.getConnection("jdbc:mysql://127.0.0.1:3306/mydb?useUnicode=true&characterEncoding=UTF-8&useServerPrepStmts=false&rewriteBatchedStatements=true","root","root");
connection.setAutoCommit(false);
PreparedStatement ps = connection.prepareStatement(
"insert into tb_user (name) values(?)");
for (int i = 0; i < stuNum; i++) {
ps.setString(1,name);
ps.addBatch();
}
ps.executeBatch();
connection.commit();
connection.close();
After testing, the use of insertion ExecutorType.BATCH manner, significantly improved performance, less than 2s able to complete the entire insert.
In summary, if MyBatis needs to bulk insert, insert ExecutorType.BATCH recommended way, if you must use the inserted words, each time you insert the records need to be controlled to about 20 to 50.
3. Program
because of concerns the JVM GC so the data should be limited to it, but in view of the large amount of data Mybatis bulk insert efficiency is not high, so the size of the data segment governance.
3.1 using less than 1W: Mybatis Bulk insert embodiment
of the JVM tuning, but the primary performance bottleneck in a batch inserting operation. Given mybatis advantage in project development, the amount of data with very small or recommend the use Mybatis.
Using less than 10W 1W greater than 3.2: JDBC + batch transaction processing
on the JVM tuning (Stack and GC settings, etc.). General Procedure 30 seconds time-consuming performance can be tolerated.
3.3 10W using the above data: the data the JDBC batch batch + + transaction
for tuning the JVM (Stack and GC settings, etc.), batch processing of the data. For batch processing requires reference to previous test data points defined batch size, mainly tuning operation time.