JDBC批量操作性能提升 和 Mysql使用规则

转自: https://blog.csdn.net/yunhua_lee/article/details/8239145https://blog.csdn.net/yunhua_lee/article/details/44776111

JDBC

当使用INSERT INTO....VALUES()语句批量插入的时候,应该使用JDBC的PreparedStatement的批量操作方法,而不是采用一条一条执行的方法。

例如(来源:http://superjavason.iteye.com/blog/255423):

如上图,代码有3个关键的处理步骤:

1)关闭自动提交

2)addBatch

3)executeBatch

使用这种方法,SQLite测试时的效果提升非常明显,从 10000/s提升到100000/s(数据仅做参考,不能作为决策依据)

需要注意的是:这种方式只适合相同结构的SQL语句批量执行,对于不同结构的SQL语句不能用这种方式,因为PreparedStatement在初始化的时候要指定sql

扫描二维码关注公众号,回复: 2750613 查看本文章

MySQL

使用上述参数后,MySQL的性能也有一定的提升,但提升不明显。经过查找,确认和如下两个参数有关:

  • rewriteBatchedStatements=true

mysql默认关闭了batch处理, 通过此参数进行打开, 这个参数可以重写向 数据库提交的SQL语句

  • useServerPrepStmts=false

如果不开启(useServerPrepStmts=false), 使用com.mysql.jdbc.PreparedStatement进行本地SQL拼装, 最后送到db上就是已经替换了?后的最终SQL

经过测试,打开这2个参数后,mysql的批处理性能从1000多提升到50000多

详细请参考:http://blog.csdn.net/whucyl/article/details/20838079

原文:http://dev.mysql.com/doc/refman/5.5/en/innodb-default-se.html
Best Practices for InnoDB Tables
If you have been using InnoDB for a long time, you already know about features like transactions and foreign keys. If not, read about them throughout this chapter. To make a long story short:

1) Specify a primary key for every table using the most frequently queried column or columns, or an auto-increment value if there isn't an obvious primary key.
指定主键:使用最频繁查询的一个或者多个列作为主键,如果没有,就使用自增id作为主键

2) Embrace the idea of joins, where data is pulled from multiple tables based on identical ID values from those tables. For fast join performance, define foreign keys on the join columns, and declare those columns with the same datatype in each table. The foreign keys also propagate deletes or updates to all affected tables, and prevent insertion of data in a child table if the corresponding IDs are not present in the parent table.
拥抱join:当数据从多个表里面按照相同的ID值取出来的时候,优先使用join,可以使用外键来提高join的性能。

3) Turn off autocommit. Committing hundreds of times a second puts a cap on performance (limited by the write speed of your storage device).
关闭autocommit:每秒提交几百次会限制性能。
(译者注:业务系统建议不要关闭,否则编程比较麻烦,批量导入数据的时候可以关闭,然后定时或者定量commit)

4) Bracket sets of related changes, logical units of work, with START TRANSACTION and COMMIT statements. While you don't want to commit too often, you also don't want to issue huge batches of INSERT, UPDATE, or DELETEstatements that run for hours without committing.
使用事务提交,避免频繁提交

5) Stop using LOCK TABLE statements. InnoDB can handle multiple sessions all reading and writing to the same table at once, without sacrificing reliability or high performance. To get exclusive write access to a set of rows, use theSELECT ... FOR UPDATE syntax to lock just the rows you intend to update.
不要使用LOCK TABLE操作,Innodb的机制能够支持高并发操作,且不损失可靠性和性能。推荐使用SELECT....FOR UPDATE

6) Enable the innodb_file_per_table option to put the data and indexes for individual tables into separate files, instead of in a single giant system tablespace. (This setting is required to use some of the other features, such as table compression and fast truncation.)
打开innodb_file_per_table选项

猜你喜欢

转载自blog.csdn.net/hemeinvyiqiluoben/article/details/81587308
今日推荐