Optimizing when inserting large amount of data into database with Java

Reprinted from: http://gooderlee.iteye.com/blog/1160949

Use jdbc to insert 100,000 records into the database, and use three methods: statement, PreparedStatement, and PreparedStatement+batch to test: 

[java]  view plain copy  

 

1. public void exec(Connection conn){      
2.   try {      
3.    //Starting time     
4.    Long beginTime = System.currentTimeMillis();      
5.    //Set manual submission      
6.    conn.setAutoCommit(false);     
7.          
8.    Statement st = conn.createStatement();      
9.     
10.    for(int i=0;i<100000;i++){      
11.       String sql="insert into t1(id) values ("+i+")";      
12.       st.executeUpdate(sql);       
13.    }      
14.         
15.    //End Time     
16.    Long endTime = System.currentTimeMillis();      
17.     
18.    System.out.println( "st:"+(endTime-beginTime)/ 1000+ "seconds"); //Calculate time      
19.     
20.    st.close();      
21.    conn.close();      
22.   } catch (SQLException e) {      
23.    e.printStackTrace ();      
24.   }       
25. }     

//2. Use PreparedStatement object 

[java]  view plain copy  

 

1. public void exec2(Connection conn){      
2.   try {      
3.    Long beginTime = System.currentTimeMillis();      
4.    conn.setAutoCommit( false); //Manual commit      
5.    PreparedStatement pst = conn.prepareStatement("insert into t1(id) values (?)");      
6.    for(int i=0;i<100000;i++){      
7.     pst.setInt(1, i);      
8.     pst.execute();         
9.    }      
10.    conn.commit();      
11.    Long endTime = System.currentTimeMillis();      
12.    System.out.println( "pst:"+(endTime-beginTime)/ 1000+ "seconds"); //Calculate time      
13.    pst.close();      
14.    conn.close();      
15.   } catch (SQLException e) {      
16.    e.printStackTrace ();      
17.   }      
18.     
19. }     

//3. Use PreparedStatement + batch processing 

[java]  view plain copy  

 

1. public void exec3(Connection conn){      
2.   try {      
3.    conn.setAutoCommit(false);      
4.    Long beginTime = System.currentTimeMillis();      
5.    //Construct the preprocessing statement     
6.    PreparedStatement pst = conn.prepareStatement("insert into t1(id) values (?)");      
7.    //10,000 cycles     
8.    for(int i=1;i<=100000;i++){         
9.     pst.setInt(1, i);      
10.     pst.addBatch();      
11.     // every 1000 commits     
12.     if(i% 1000== 0){ //You can set different sizes; such as 50, 100, 500, 1000, etc.      
13.      pst.executeBatch();      
14.      conn.commit();      
15.      pst.clearBatch();      
16.     }      
17.    }     
18.    Long endTime = System.currentTimeMillis();      
19.    System.out.println("pst+batch："+(endTime-beginTime)/1000+"秒");      
20.    pst.close();      
21.    conn.close();      
22.   } catch (SQLException e) {      
23.    e.printStackTrace ();      
24.   }      
25. }     

Tested in Oracle 10g, the results: 
1. Using statement takes 142 seconds; 
2. Using PreparedStatement takes 56 seconds; 
3. Using PreparedStatement + batch processing: 
   a. 50 inserts once, it takes 5 seconds; 
    b. It takes 2 seconds to insert 100 records once; 
    c. It takes 1 second to insert more than 1000 records once; 
from the above, it can be concluded that when using jdbc to insert data in large batches, the third method (PreparedStatement + batch processing) is obviously used. ) for better performance.

-------------------------------------------------------------------------------------------------------------------------------

1. When inserting a large amount of data in the ordinary way, the processing speed is quite slow.   
2. */   
3. PreparedStatement ps = null;   
4. //loop 10000 times   
5. for(int i = 0; i < 100000; i++) {   
6.     ps = con.prepareStatement(sql);   
7.     ps.executeUpdate();   
8. }  

 

1. Method 2: Cache the data in the object by means of addBatch(), and submit it by executing the executeBatch(); method at the end, so the speed will be much faster!   
2. */   
3. PreparedStatement ps = con.prepareStatement(sql);   
4. for(int i = 0; i < 100000; i++) {   
5.     ps.setString(1, "1");   
6.     ps.setString(2, "2");   
7.     ps.addBatch();   
8. }   
9. ps.executeBatch();