background
A project, after obtaining data from Kafka, generates insert, update, delete statements through business processing, and executes them in the Oracle database. The project initially uses a single sql statement to execute the sql statement in a non-precompiled and submitted manner. With the increase of kafka data, Oracle's performance of executing SQL statements deteriorated, and finally led to the collapse of the entire Oracle database, unable to execute any SQL statements. Hence the need for optimization. The final optimization solution adopted was batch submission + SQL precompilation to solve the problem. Let's discuss the impact of batch submission and SQL precompilation on performance improvement.
SQL precompilation and batch submission
When it comes to SQL precompilation, the first thing that comes to mind is to prevent SQL injection. How does SQL compilation help improve SQL execution performance? The blogger has consulted a lot of information, and most of them use the following figure to illustrate:
When a client sends a SQL statement to the server, the server always needs to verify whether the syntax of the SQL statement is correct. Then compile the SQL statement into an executable function, and finally execute the SQL statement. Among them, checking the syntax and compiling may take more time than executing the SQL statement.
In the Oracle database and the Mysql database, the mechanisms for implementing SQL precompilation are different. The following experiments will compare the difference in the performance improvement of SQL precompilation for the two databases.
In the batch submission in jdbc, the precompilation (PrepareStatement) must be the sql statement of the same structure (the sql statement is exactly the same, and the parameters are different) before it can be submitted in batches. SQL statements with different structures cannot be submitted in the same batch. Non-precompiled submission (Statement) can submit SQL statements with different structures in batches.
Based on the above mechanism, precompiled SQL can only be submitted in batches with the same statement. If there are few SQL statements with the same structure and most of them are SQL statements with different structures, then whether to submit precompiled in batches with high performance or to submit non-precompiled in batches High performance, how to choose? Solve the doubts through the following experiment.
Oracle Data Experiment
Use the native jdbc to connect to the oracle database, and perform non-precompiled single submission, precompiled single submission, non-precompiled batch submission, and precompiled batch submission for comparison.
The test table structure is as follows:
insert statement
Non-precompiled single commit
code show as below:
/**
* 非预编译，单条提交，提交5000次,insert
* @throws Exception
*/
public static void test1_1() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
long start=System.currentTimeMillis();
for(int i=0;i<10000;i++){
String sql="";
sql="INSERT INTO ceshi (\n" +
"\t name,\n" +
"\t age,\n" +
"\t sex,\n" +
"\t school,\n" +
"\t birth,\n" +
"\t money,\n" +
"\t createtime,\n" +
"\t remark \n" +
")\n" +
"VALUES\n" +
"\t(\n" +
"\t\t '张三',\n" +
"\t\t '18',\n" +
"\t\t '男',\n" +
"\t\t '测试测试',\n" +
"\t\t to_date('2023/03/11','yyyy-MM-dd'),\n" +
"\t\t '10.22',\n" +
"\t\t to_date('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS'),\n" +
"\t\t'测试11111111测试222222222测试'\n" +
"\t)";
boolean result = stat.execute(sql);
}
System.out.println("test1_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
Insert 5000 pieces of data, execute three times, the result is as follows:
test1_1 execution time: 806
test1_1 execution time: 749
test1_1 execution time: 762
Precompile a single commit
code show as below:
/**
* 预编译,单条提交，提交5000次，insert
*/
public static void test1_2() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
//执行 sql 语句的对象
String sql="INSERT INTO ceshi (\n" +
"\t name,\n" +
"\t age,\n" +
"\t sex,\n" +
"\t school,\n" +
"\t birth,\n" +
"\t money,\n" +
"\t createtime,\n" +
"\t remark \n" +
")\n" +
"VALUES\n" +
"\t(\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ? \n" +
"\t)";
PreparedStatement stat = com.prepareStatement(sql);
long start=System.currentTimeMillis();
for(int i=0;i<10000;i++){
Date date = new Date(System.currentTimeMillis());
Timestamp tt = new Timestamp(date.getTime());
stat.setObject(1,"张三"+i);
stat.setObject(2,20);
stat.setObject(3,"男"+i);
stat.setObject(4,"测试ceshi"+i);
//stat.setObject(5,"to_date('2023/03/11','yyyy-MM-dd')");
stat.setObject(5, date);
stat.setObject(6,10);
// stat.setObject(7,"TO_TIMESTAMP('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS')");
//stat.setObject(7,Calendar.getInstance().getTime());
stat.setObject(7,tt);
stat.setObject(8,"测试11111111测试222222222测试"+i);
boolean result = stat.execute();
}
System.out.println("test1_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
Insert 5000 pieces of data, the results are as follows:
test1_2 execution time: 771
test1_2 execution time: 787
test1_2 execution time: 780
Non-precompiled batch submission
code show as below:
/**
* 非预编译批量提交 insert 5000条
*/
public static void test2_1() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
for(int i=0;i<10000;i++){
String sql="";
sql="INSERT INTO ceshi (\n" +
"\t name,\n" +
"\t age,\n" +
"\t sex,\n" +
"\t school,\n" +
"\t birth,\n" +
"\t money,\n" +
"\t createtime,\n" +
"\t remark \n" +
")\n" +
"VALUES\n" +
"\t(\n" +
"\t\t '张三',\n" +
"\t\t '18',\n" +
"\t\t '男',\n" +
"\t\t '测试测试',\n" +
"\t\t to_date('2023/03/11','yyyy-MM-dd'),\n" +
"\t\t '10.22',\n" +
"\t\t to_date('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS'),\n" +
"\t\t'测试11111111测试222222222测试'\n" +
"\t)";
stat.addBatch(sql);
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("test2_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
Submit 5,000 items in batches at one time, and the results are as follows:
test2_1 execution time: 779
test2_1 execution time: 834
test2_1 execution time: 823
Precompiled Batch Submit
code show as below:
/**
* 预编译批量提交 insert 5000条
*/
public static void test2_2() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
//执行 sql 语句的对象
String sql="INSERT INTO ceshi (\n" +
"\t name,\n" +
"\t age,\n" +
"\t sex,\n" +
"\t school,\n" +
"\t birth,\n" +
"\t money,\n" +
"\t createtime,\n" +
"\t remark \n" +
")\n" +
"VALUES\n" +
"\t(\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ?,\n" +
"\t\t ? \n" +
"\t)";
PreparedStatement stat = com.prepareStatement(sql);
for(int i=0;i<10000;i++){
Date date = new Date(System.currentTimeMillis());
Timestamp tt = new Timestamp(date.getTime());
stat.setObject(1,"张三"+i);
stat.setObject(2,20);
stat.setObject(3,"男"+i);
stat.setObject(4,"测试ceshi"+i);
//stat.setObject(5,"to_date('2023/03/11','yyyy-MM-dd')");
stat.setObject(5, date);
stat.setObject(6,10);
// stat.setObject(7,"TO_TIMESTAMP('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS')");
//stat.setObject(7,Calendar.getInstance().getTime());
stat.setObject(7,tt);
stat.setObject(8,"测试11111111测试222222222测试"+i);
stat.addBatch();
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("test2_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
Submit 5,000 items at one time, and the execution results are as follows:
test2_2 execution time: 34
test2_2 execution time: 59
test2_2 execution time: 36
insert...values(),() insert multiple statements
In oracle, the sql syntax for an insert to insert multiple values is:
INSERT ALL
INTO ceshi(name,age,sex,school,birth,money,createtime,remark) values('测试1',19,'女','xuexiao',to_date('2023/03/11','yyyy-MM-dd'),2023,to_date('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS'),'测试111')
INTO ceshi(name,age,sex,school,birth,money,createtime,remark) values('测试2',19,'女','xuexiao',to_date('2023/03/11','yyyy-MM-dd'),2023,to_date('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS'),'测试111')
INTO ceshi(name,age,sex,school,birth,money,createtime,remark) values('测试3',19,'女','xuexiao',to_date('2023/03/11','yyyy-MM-dd'),2023,to_date('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS'),'测试111')
SELECT 1 FROM DUAL;
Test this statement to insert 1000 pieces of data at one time, the code is as follows:
public static void test3_1() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
String sql="INSERT ALL ";
for(int i=0;i<1000;i++){
sql+=" INTO ceshi (\n" +
"\t name,\n" +
"\t age,\n" +
"\t sex,\n" +
"\t school,\n" +
"\t birth,\n" +
"\t money,\n" +
"\t createtime,\n" +
"\t remark \n" +
")\n" +
"VALUES\n" +
"\t(\n" +
"\t\t '张三',\n" +
"\t\t '18',\n" +
"\t\t '男',\n" +
"\t\t '测试测试',\n" +
"\t\t to_date('2023/03/11','yyyy-MM-dd'),\n" +
"\t\t '10.22',\n" +
"\t\t to_date('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS'),\n" +
"\t\t'测试11111111测试222222222测试'\n" +
"\t) ";
}
sql+=" SELECT 1 FROM DUAL";
long start=System.currentTimeMillis();
stat.execute(sql);
System.out.println("test3_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
The execution results are as follows:
test3_1 execution time: 1748
in conclusion
A summary of the above results is as follows:
Non-precompiled, single submission, 5000 submissions
test1_1 execution time: 806
test1_1 execution time: 749
test1_1 execution time: 762Precompile, single submission, 5000 submissions
test1_2 execution time: 771
test1_2 execution time: 787
test1_2 execution time: 780Non-precompiled batch submission insert 5000
test2_1 execution time: 779
test2_1 execution time: 834
test2_1 execution time: 823
precompiled batch submission insert 5000
test2_2 execution time: 34
test2_2 execution time: 59
test2_2 execution time: 36
insert...values statement once Sexual insertion of 1000
test3_1 execution time: 1748
From the above results, it can be seen that the performance of insert...values...multiple values is the worst. The performance of non-precompiled single submission, precompiled single submission, and non-precompiled batch submission are all in the same order of magnitude. Precompiled bulk commits perform the fastest. Therefore, when inserting data in large batches into the Oracle database, precompiled batch submission is used to achieve the fastest performance.
update statement
Non-precompiled single commit
code show as below:
/**
* 非预编译单条update提交,5000条
*/
public static void test4_1() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
Random random=new Random();
//执行 sql 语句的对象
long start=System.currentTimeMillis();
for(int i=0;i<10000;i++){
String sql="";
sql="update ceshi set age=11"+i+",sex='未知"+i+"',school='修改"+i+"',birth=to_date('2023/03/11','yyyy-MM-dd') where id = "+(35000+i);//*random.nextInt(30000)
// System.out.println(sql);
boolean result = stat.execute(sql);
}
System.out.println("test4_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
Execute 5000 entries, the results are as follows:
test4_1 execution time: 14881
Execute 10,000 entries, the results are as follows:
test4_1 execution time: 32295
Precompile a single commit
code show as below:
/**
* 预编译单条update提交,5000条
* @throws Exception
*/
public static void test4_2() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
//执行 sql 语句的对象
String sql="update ceshi set age=?,sex=?,school=?,birth=? where id = ?";
PreparedStatement stat = com.prepareStatement(sql);
long start=System.currentTimeMillis();
for(int i=0;i<10000;i++){
Date date = new Date(System.currentTimeMillis());
Timestamp tt = new Timestamp(date.getTime());
stat.setObject(1,i);
stat.setObject(2,"男"+i);
stat.setObject(3,"测试ceshi"+i);
//stat.setObject(5,"to_date('2023/03/11','yyyy-MM-dd')");
stat.setObject(4, date);
stat.setObject(5,15000+i);
// stat.setObject(7,"TO_TIMESTAMP('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS')");
//stat.setObject(7,Calendar.getInstance().getTime());
boolean result = stat.execute();
}
System.out.println("test4_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
Execute 5000 entries, the results are as follows:
test4_2 execution time: 11430
Execute 10,000 entries, the results are as follows:
test4_2 execution time: 25123
Non-precompiled batch submission
code show as below:
/**
* 非预编译，批量提交5000条
*/
public static void test5_1() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
for(int i=0;i<10000;i++){
String sql="update ceshi set age=11"+i+",sex='未知"+i+"',school='修改"+i+"',birth=to_date('2023/03/11','yyyy-MM-dd') where id = "+(50000+i);//*random.nextInt(30000)
stat.addBatch(sql);
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("test5_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
Execute 5000 entries, the results are as follows:
test5_1 execution time: 15733
Execute 10,000 entries, the results are as follows:
test5_1 execution time: 31539
Precompiled Batch Submit
code show as below:
/**
* 预编译批量提交5000条
*/
public static void test5_2() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
//执行 sql 语句的对象
String sql="update ceshi set age=?,sex=?,school=?,birth=? where id = ?";
PreparedStatement stat = com.prepareStatement(sql);
for(int i=0;i<10000;i++){
Date date = new Date(System.currentTimeMillis());
Timestamp tt = new Timestamp(date.getTime());
stat.setObject(1,i);
stat.setObject(2,"男"+i);
stat.setObject(3,"测试ceshi"+i);
//stat.setObject(5,"to_date('2023/03/11','yyyy-MM-dd')");
stat.setObject(4, date);
stat.setObject(5,30000+i);
// stat.setObject(7,"TO_TIMESTAMP('2023-11-11 11:11:11','yyyy-mm-dd HH:MI:SS')");
//stat.setObject(7,Calendar.getInstance().getTime());
stat.addBatch();
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("test5_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
Execute 5000 entries, the results are as follows:
test5_2 execution time: 8808
Execute 10,000 entries, the results are as follows:
test5_2 execution time: 17672
in conclusion
The summary update experiment results are as follows:
Non-precompiled single update submission, 5000 items:
test4_1 execution time: 14881
non-precompiled single update submission, 10000 items:
test4_1 execution time: 32295
precompiled single update submissions, 5000 items:
test4_2 execution time: 11430
precompiled single update submissions, 10000 entries:
test4_2 execution time: 25123
non-precompiled batch submission update, 5000 entries:
test5_1 execution time: 15733
non-precompiled batch submission update, 10000 entries:
test5_1 execution time: 31539
precompiled batch submission update, 5000 entries:
test5_2 execution time :8808
precompiled batch submission update, 10000 items:
test5_2 Execution time: 17672
From the above results, we can see that the performance of precompiled single submission is higher than that of non-precompiled single submission. This proves that precompilation does reduce the time for repeated SQL parsing and verification, and improves performance. But overall, the effect of precompilation on non-precompilation performance is not particularly obvious.
The performance of non-precompiled batch submission is basically the same as that of non-batch submission single submission, which means that the performance improvement of batch submission is not very obvious.
The performance of precompiled batch submission is significantly better than non-precompiled single submission and non-precompiled batch submission, and the performance is twice as fast. It can be seen that in Oracle, precompilation greatly improves performance. In the case of precompilation, the performance improvement of batch submission is somewhat obvious compared with that of single submission.
Therefore, in the case of Oracle database update, SQL precompilation is the first factor that affects performance, and batch submission is the second factor. Moreover, the execution performance of the update statement is much lower than that of the insert.
delete statement
code show as below:
/**
* 非预编译单条delete,5000条
*/
public static void test6_1() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
long start=System.currentTimeMillis();
for(int i=0;i<5000;i++){
String sql="";
sql="delete from ceshi where id="+i;
boolean result = stat.execute(sql);
}
System.out.println("test6_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
/**
* 预编译单条delet,5000条
*/
public static void test6_2() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
//执行 sql 语句的对象
String sql="delete from ceshi where id = ?";
PreparedStatement stat = com.prepareStatement(sql);
long start=System.currentTimeMillis();
for(int i=0;i<5000;i++){
Date date = new Date(System.currentTimeMillis());
Timestamp tt = new Timestamp(date.getTime());
stat.setObject(1,i+10000);
boolean result = stat.execute();
}
System.out.println("test6_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
/**
* 非预编译批量提交,5000条
*/
public static void test7_1() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
for(int i=0;i<5000;i++){
String sql= sql="delete from ceshi where id="+(i+20000);
stat.addBatch(sql);
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("test7_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
/**
* 预编译批量提交,5000条
*/
public static void test7_2() throws Exception{
Class.forName("oracle.jdbc.driver.OracleDriver");
Connection com = DriverManager.getConnection("jdbc:oracle:thin:@localhost:1521/orcl?useUnicode=true&characterEncoding=utf8", "demo", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
//执行 sql 语句的对象
String sql="delete from ceshi where id = ?";
PreparedStatement stat = com.prepareStatement(sql);
for(int i=0;i<5000;i++){
Date date = new Date(System.currentTimeMillis());
Timestamp tt = new Timestamp(date.getTime());
stat.setObject(1,i+5000);
stat.addBatch();
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("test7_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
The execution results are as follows:
delete statement:
non-precompiled single delete, 5000:
test6_1 execution time: 15559
Precompile a single delete, 5000:
test6_2 Execution time: 12477
Non-precompiled batch submission of 5000 items:
Precompiled batch submission, 5000 items:
test7_1 Execution time: 15599
Precompiled batch submission, 5000 entries:
test7_2 Execution time: 7433
in conclusion
From the above experimental data, it can be seen that the delete statement is similar to the update statement, and the non-precompiled single and batch performance are the same. The performance of precompilation is significantly improved, and the performance of precompilation batch submission is better.
Oracle Overall Conclusion
For Oracle database, executing insert statement is faster than update statement and delete statement. And precompiled SQL has an effect on improving SQL execution performance. For batch submissions, there is not much difference in performance between non-precompiled batch submissions and single submissions. The performance improvement of precompiled batch submission is more obvious than that of precompiled single submission. Therefore, to improve the execution performance of Oracle database sql, precompilation + batch submission is the best optimization solution.
MYSQL experimental data
Let's look at the batch submission and precompiled SQL of the MYSQL database. Is the conclusion consistent with the Oracle database? It is important to note that in MYSQL, the precompilation and batch submission functions are disabled by default. You must add the following parameters to the url of the jdbc connection database to enable the precompilation and batch submission functions:
rewriteBatchedStatements=true to enable batch submission
useServerPrepStmts=true to enable SQL precompilation
cachePrepStmts=true to enable precompiled cache
The above three parameters are not explained here. This article mainly compares the performance of SQL execution.
insert statement
Precompiled/non-precompiled single commit
code show as below:
public static void test1_1() throws Exception{
Class.forName("com.mysql.cj.jdbc.Driver");
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?useServerPrepStmts=true&cachePrepStmts=true", "root", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
long start=System.currentTimeMillis();
for(int i=0;i<5000;i++){
String sql="";
sql="INSERT INTO `demo`.`ceshi` (\n" +
"\t`name`,\n" +
"\t`age`,\n" +
"\t`sex`,\n" +
"\t`school`,\n" +
"\t`birth`,\n" +
"\t`money`,\n" +
"\t`createtime`,\n" +
"\t`remark`\n" +
")\n" +
"VALUES\n" +
"\t(\n" +
"\t\t'张三',\n" +
"\t\t'18',\n" +
"\t\t'男',\n" +
"\t\t'测试测试',\n" +
"\t\t'2023-03-07',\n" +
"\t\t'10.22',\n" +
"\t\t'2023-03-07 23:05:11',\n" +
"\t\t'测试11111111测试222222222测试'\n" +
"\t)";
boolean result = stat.execute(sql);
}
System.out.println("test1_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
public static void test1_2() throws Exception{
//
Class.forName("com.mysql.cj.jdbc.Driver");
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?useServerPrepStmts=true&cachePrepStmts=true", "root", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
String sql="INSERT INTO ceshi(`name`,`age`,`sex`,`school`,`birth`,`money`,`createtime`,`remark`) VALUES(?,?,?,?,?,?,?,?)";
System.out.println(sql);
PreparedStatement stat = com.prepareStatement(sql);
//执行 sql 语句的对象
long start=System.currentTimeMillis();
for(int i=0;i<5000;i++){
stat.setObject(1,"张三"+i);
stat.setObject(2,"20"+i);
stat.setObject(3,"男"+i);
stat.setObject(4,"测试ceshi"+i);
stat.setObject(5,"2023-03-07");
stat.setObject(6,"10.22");
stat.setObject(7,"2023-03-07 23:05:11");
stat.setObject(8,"测试11111111测试222222222测试"+i);
boolean result = stat.execute();
}
System.out.println("test1_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
The execution results are as follows:
Non-precompiled, single submission, 5000 submissions, insert:
test1_1 execution time: 12177
test1_1 execution time: 10684
non-precompiled, single submission, 10000 submissions, insert:
test1_1 execution time: 21078
test1_1 execution time: 21581
precompilation, single Submit, submit 5000 times, insert
test1_2 execution time: 10660
test1_2 execution time: 10885
precompile, single submission, submit 10000 times, insert
test1_2 execution time: 21487
test1_2 execution time: 21710
Conclusion: For the insert of MYSQL, the performance of precompilation and non-precompilation is as poor for a single submission of SQL.
Precompiled/non-precompiled batch submission
code show as below:
public static void test2_1() throws Exception{
Class.forName("com.mysql.cj.jdbc.Driver");
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?rewriteBatchedStatements=true", "root", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
for(int i=0;i<10000;i++){
String sql="";
sql="INSERT INTO `demo`.`ceshi` (\n" +
"\t`name`,\n" +
"\t`age`,\n" +
"\t`sex`,\n" +
"\t`school`,\n" +
"\t`birth`,\n" +
"\t`money`,\n" +
"\t`createtime`,\n" +
"\t`remark`\n" +
")\n" +
"VALUES\n" +
"\t(\n" +
"\t\t'张三',\n" +
"\t\t'18',\n" +
"\t\t'男',\n" +
"\t\t'测试测试',\n" +
"\t\t'2023-03-07',\n" +
"\t\t'10.22',\n" +
"\t\t'2023-03-07 23:05:11',\n" +
"\t\t'测试11111111测试222222222测试'\n" +
"\t)";
stat.addBatch(sql);
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("test2_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
public static void test2_2() throws Exception{
//&useServerPrepStmts=true
Class.forName("com.mysql.cj.jdbc.Driver");
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?rewriteBatchedStatements=true&useServerPrepStmts=true", "root", "AsDf!123");
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
String sql="INSERT INTO ceshi(`name`,`age`,`sex`,`school`,`birth`,`money`,`createtime`,`remark`) VALUES(?,?,?,?,?,?,?,?)";
System.out.println(sql);
PreparedStatement stat = com.prepareStatement(sql);
//执行 sql 语句的对象
for(int i=0;i<5000;i++){
stat.setObject(1,"张三"+i);
stat.setObject(2,"20"+i);
stat.setObject(3,"男"+i);
stat.setObject(4,"测试ceshi"+i);
stat.setObject(5,"2023-03-07");
stat.setObject(6,"10.22");
stat.setObject(7,"2023-03-07 23:05:11");
stat.setObject(8,"测试11111111测试222222222测试"+i);
stat.addBatch();
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("test2_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
The execution results are as follows:
Submit 5000 inserts in batches without pre-compilation:
test2_1 execution time: 9781
test2_1 execution time: 10137
pre-compilation batches insert 5000 entries: (do not add useServerPrepStmts parameter, do not enable SQL precompilation)
test2_2 execution time: 96
test2_2 execution time: 93
pre-compiled Compile and submit 5000 inserts in batches: (add useServerPrepStmts parameter to enable SQL precompilation)
test2_2 execution time: 173
test2_2 execution time: 154
Conclusion: The performance of MYSQL precompiled and batch submitted insert statements is much higher than that of non-precompiled. Non-precompiled batch commits have slightly improved performance over single commits (compiled/non-precompiled).
There is a problem with the above results. Why is the performance of non-precompiled batch submissions higher than that of precompilation without precompilation enabled? According to common sense, if precompilation is not enabled, it is equivalent to batch submission without precompilation, and the performance should not be higher than that with precompilation enabled. With this doubt, I checked the MYSQL log file. It is found that when SQL precompilation is not enabled, MYSQL splices the insert statement of PrepareStatement into a multi-data insert in the format of insert ... values(),()...(). After the precompilation function is enabled, MYSQL will first prepare and precompile the insert sql, and then execute the insert statement. In this way, the efficiency is slower. Therefore, in MYSQL insert, insert...values(),()...() is the most efficient.
update statement
code show as below:
/**
* 预编译与非预编译update语句比较---阈值
*/
public static void test5_1() throws Exception{
Class.forName("com.mysql.cj.jdbc.Driver");
System.out.println(000);
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?rewriteBatchedStatements=true", "root", "AsDf!123");
System.out.println(666);
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
//执行 sql 语句的对象
System.out.println(111);
for(int i=0;i<5000;i++){
String sql="UPDATE `demo`.`ceshi`\n" +
" set `name` = '张三"+i+"',\n" +
" `age` = '18"+i+"',\n" +
" `sex` = '男"+i+"',\n" +
" `school` = '测试测试"+i+"',\n" +
" `birth` = '2023-03-07',\n" +
" `money` = '10.22',\n" +
" `createtime` = '2023-03-07 23:05:11',\n" +
" `remark` = '测试11111111测试222222222测试'\n" +
"WHERE\n" +
"\t(`id` = '"+(141646+i)+"')";
stat.addBatch(sql);
}
System.out.println(222);
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("第一次test5_1执行时间:"+(System.currentTimeMillis()-start));
stat.clearBatch();
for(int i=0;i<5000;i++){
String sql="UPDATE `demo`.`ceshi`\n" +
" set `name` = '张三"+i+"',\n" +
" `age` = '18"+i+"',\n" +
" `sex` = '男"+i+"',\n" +
" `school` = '测试测试"+i+"',\n" +
" `birth` = '2023-03-07',\n" +
" `money` = '10.22',\n" +
" `createtime` = '2023-03-07 23:05:11',\n" +
" `remark` = '测试11111111测试222222222测试'\n" +
"WHERE\n" +
"\t(`id` = '"+(160646+i)+"')";
stat.addBatch(sql);
}
System.out.println(222);
start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("第二次test5_1执行时间:"+(System.currentTimeMillis()-start));
stat.clearBatch();
for(int i=0;i<5000;i++){
String sql="UPDATE `demo`.`ceshi`\n" +
" set `name` = '张三"+i+"',\n" +
" `age` = '18"+i+"',\n" +
" `sex` = '男"+i+"',\n" +
" `school` = '测试测试"+i+"',\n" +
" `birth` = '2023-03-07',\n" +
" `money` = '10.22',\n" +
" `createtime` = '2023-03-07 23:05:11',\n" +
" `remark` = '测试11111111测试222222222测试'\n" +
"WHERE\n" +
"\t(`id` = '"+(182316+i)+"')";
stat.addBatch(sql);
}
System.out.println(222);
start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("第三次test5_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
public static void test5_2() throws Exception{
//&useServerPrepStmts=true&cachePrepStmts=true
Class.forName("com.mysql.cj.jdbc.Driver");
System.out.println(000);
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?rewriteBatchedStatements=true", "root", "AsDf!123");
System.out.println(111);
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
String sql="UPDATE ceshi\n" +
"SET \n" +
" `name` = ?," +
" `age` = ?," +
" `sex` = ?," +
" `school` = ?," +
" `birth` = ?," +
" `money` = ?," +
" `createtime` = ?," +
" `remark` = ?" +
" WHERE\n" +
"\t`id` = ?";
PreparedStatement stat = com.prepareStatement(sql);
//执行 sql 语句的对象
for(int i=0;i<5000;i++){
stat.setObject(1,"张三"+i);
stat.setObject(2,"191"+i);
stat.setObject(3,"1男1"+i);
stat.setObject(4,"测试ceshi1"+i);
stat.setObject(5,"2023-03-07");
stat.setObject(6,"10.2211");
stat.setObject(7,"2023-03-07 23:05:11");
stat.setObject(8,"测试11111111测试222222222测试"+i);
stat.setObject(9,189305+i);
stat.addBatch();
}
long start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("第一次test5_2执行时间:"+(System.currentTimeMillis()-start));
stat.clearBatch();
//执行 sql 语句的对象
for(int i=0;i<5000;i++){
stat.setObject(1,"张三"+i);
stat.setObject(2,"191"+i);
stat.setObject(3,"1男1"+i);
stat.setObject(4,"测试ceshi1"+i);
stat.setObject(5,"2023-03-07");
stat.setObject(6,"10.2211");
stat.setObject(7,"2023-03-07 23:05:11");
stat.setObject(8,"测试11111111测试222222222测试"+i);
stat.setObject(9,402287+i);
stat.addBatch();
}
start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("第二次test5_2执行时间:"+(System.currentTimeMillis()-start));
stat.clearBatch();
//执行 sql 语句的对象
for(int i=0;i<5000;i++){
stat.setObject(1,"张三"+i);
stat.setObject(2,"191"+i);
stat.setObject(3,"1男1"+i);
stat.setObject(4,"测试ceshi1"+i);
stat.setObject(5,"2023-03-07");
stat.setObject(6,"10.2211");
stat.setObject(7,"2023-03-07 23:05:11");
stat.setObject(8,"测试11111111测试222222222测试"+i);
stat.setObject(9,602287+i);
stat.addBatch();
}
start=System.currentTimeMillis();
stat.executeBatch();
System.out.println("第三次test5_2执行时间:"+(System.currentTimeMillis()-start));
stat.clearBatch();
com.close();
stat.close();
com.close();
}
/**
* 预编译与非预编译单条提交 update语句测试
*/
public static void test6_1() throws Exception{
Class.forName("com.mysql.cj.jdbc.Driver");
System.out.println(000);
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?rewriteBatchedStatements=true", "root", "AsDf!123");
System.out.println(666);
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
long start=System.currentTimeMillis();
//执行 sql 语句的对象
for(int i=0;i<5000;i++){
// long ss=System.currentTimeMillis();
String sql="UPDATE `demo`.`ceshi`\n" +
" set `name` = '张三"+i+"',\n" +
" `age` = '18"+i+"',\n" +
" `sex` = '男"+i+"',\n" +
" `school` = '测试测试"+i+"',\n" +
" `birth` = '2023-03-07',\n" +
" `money` = '10.22',\n" +
" `createtime` = '2023-03-07 23:05:11',\n" +
" `remark` = '测试11111111测试222222222测试'\n" +
"WHERE\n" +
"\t(`id` = '"+(12426+i)+"')";
stat.execute(sql);
//System.out.println("test6_1第"+i+"次执行时间:"+(System.currentTimeMillis()-ss));
}
System.out.println("总共test6_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
public static void test6_2() throws Exception{
//&useServerPrepStmts=true
Class.forName("com.mysql.cj.jdbc.Driver");
System.out.println(000);
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?rewriteBatchedStatements=true", "root", "AsDf!123");
System.out.println(111);
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
String sql="UPDATE ceshi\n" +
"SET \n" +
" `name` = ?," +
" `age` = ?," +
" `sex` = ?," +
" `school` = ?," +
" `birth` = ?," +
" `money` = ?," +
" `createtime` = ?," +
" `remark` = ?" +
" WHERE\n" +
"\t`id` = ?";
PreparedStatement stat = com.prepareStatement(sql);
//执行 sql 语句的对象
long start=System.currentTimeMillis();
for(int i=0;i<5000;i++){
// long ss=System.currentTimeMillis();
stat.setObject(1,"张三"+i);
stat.setObject(2,"191"+i);
stat.setObject(3,"1男1"+i);
stat.setObject(4,"测试ceshi1"+i);
stat.setObject(5,"2023-03-07");
stat.setObject(6,"10.2211");
stat.setObject(7,"2023-03-07 23:05:11");
stat.setObject(8,"测试11111111测试222222222测试"+i);
stat.setObject(9,567299+i);
stat.execute();
// System.out.println("test6_2第"+i+"次执行时间:"+(System.currentTimeMillis()-ss));
}
System.out.println("第一次test6_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
The execution results are as follows:
Non-precompiled single update submission, 5000 items:
total test6_1 execution time: 10292
total test6_1 execution time: 10947
precompiled single update submission, 5000 items:
first test6_2 execution time: 11026
first test6_2 execution time: 13048
non-precompiled Submit update 5000 in batches:
first test5_1 execution time: 9051
second test5_1 execution time: 7818
third test5_1 execution time: 6747
pre-compiled batch submit update 5000:
first test5_2 execution time: 11259
second test5_2 Execution time: 11099
The third test5_2 execution time: 10489
Conclusion: Whether it is a single submission or a batch submission, the performance of precompilation and non-precompilation is similar. Batch commits perform slightly better than single commits. For MYSQL update, precompilation cannot improve performance. Batch submission improves performance slightly.
delete statement
code show as below:
/**
* 删除语句 批量提交 预编译与非预编译比较
*/
public static void test7_1() throws Exception{
Class.forName("com.mysql.cj.jdbc.Driver");
System.out.println(000);
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?rewriteBatchedStatements=true&useServerPrepStmts=true&cachePrepStmts=true", "root", "AsDf!123");
System.out.println(666);
com.setAutoCommit(false);
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
Statement stat = com.createStatement();
long start=System.currentTimeMillis();
//执行 sql 语句的对象
for(int i=0;i<10000;i++){
// long ss=System.currentTimeMillis();
String sql="DELETE from `demo`.`ceshi`\n" +
"WHERE\n" +
"\t(`id` = '"+(11150+i)+"')";
stat.addBatch(sql);
//System.out.println("test6_1第"+i+"次执行时间:"+(System.currentTimeMillis()-ss));
}
stat.executeBatch();
com.commit();
System.out.println("总共test7_1执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
public static void test7_2() throws Exception{
Class.forName("com.mysql.cj.jdbc.Driver");
System.out.println(000);
Connection com = DriverManager.getConnection("jdbc:mysql://localhost:3306/demo?rewriteBatchedStatements=true&useServerPrepStmts=true&cachePrepStmts=true", "root", "AsDf!123");
System.out.println(111);
com.setAutoCommit(false);
// DriverManager 注册驱动
// Connection 数据库连接对象 url（指定连接的路径 语法：“jdbc:mysql://ip地址:端口号/数据库名称”）
String sql="delete from ceshi\n" +
" WHERE\n" +
"\t`id` = ?";
PreparedStatement stat = com.prepareStatement(sql);
//执行 sql 语句的对象
long start=System.currentTimeMillis();
for(int i=0;i<10000;i++){
// long ss=System.currentTimeMillis();
stat.setObject(1,32260+i);
stat.addBatch();
// System.out.println("test6_2第"+i+"次执行时间:"+(System.currentTimeMillis()-ss));
}
stat.executeBatch();
com.commit();
System.out.println("第一次test7_2执行时间:"+(System.currentTimeMillis()-start));
com.close();
stat.close();
com.close();
}
The execution results are as follows:
Single submission, 100 submissions:
total test7_1 execution time: 4116 (non-precompiled)
first test7_2 execution time: 4303 (precompiled)
batch submission, 100 submissions:
total test7_1 execution time: 164 (non-precompiled)
first Times test7_2 execution time: 158 (precompiled)
Conclusion: Precompilation will not improve delete performance, but batch submission will improve performance.
MYSQL conclusion
For MYSQL, precompilation can improve SQL performance only in insert statements. However, the best way to improve insert performance in large batches is the insert...values(),()...() statement, and precompilation is not the first choice. In update statements, precompilation does not improve SQL performance. Batch submission will improve some execution performance. In MYSQL, the performance of insert...values(),()...() and batch delete is fast, and the execution performance of update statement is also very slow.
final conclusion
Batch submission and SQL precompilation are quite different in Oracle database and Mysql database.
In Oracle, precompilation greatly improves SQL execution performance. On the premise of precompilation, batch submission will further improve performance.
In Mysql, precompilation has no effect on the performance improvement of SQL execution, it may be just to prevent SQL injection. insert...values(),()...() inserts with the best performance. Batch submission of update and delete can improve performance.