Solve the problem of slow speed of MySQL importing large amount of data

1. Select the way to export data

        First of all, I would like to give a suggestion, try to use the command line tools that come with MySQL to export and import data, and do not use graphical tools such as Navicat and workbench. Especially when there is a large amount of data, using the command line tools that come with MySQL to export and import is several times faster than using graphical tools such as Navicat, and it is easy to get stuck when using graphical tools such as Navicat to perform operations with large amounts of data. The following briefly introduces how to use the command line tools that come with MySQL to import and export.

# 导出整个实例
mysqldump -uroot -pxxxxxx --all-databases > all_database.sql

# 导出指定库
mysqldump -uroot -pxxxxxx --databases testdb > testdb.sql

# 导出指定表
mysqldump -uroot -pxxxxxx testdb test_tb > test_tb.sql

# 导入指定SQL文件 （指定导入testdb库中）
mysql -uroot -pxxxxxx testdb < testdb.sql

        Most of the imported SQL script content is to build the database and table first, and then insert the data. The longest time-consuming should be the insert to insert the data. In order to reduce the file size, it is recommended to use the extended insert method, that is, to insert multiple rows together in batches, like this: insert into table_name values ​​(),(),(),...,(); . Inserting using extensions is much smaller than inserting one by one, and the file size is much smaller, and the insertion speed is several times faster.

        By default, the files exported by mysqldump use the method of batch insertion. When exporting, you can use the --skip-extended-insert parameter to insert one by one.

2. Try to modify the parameters to speed up the import

        In MySQL, there is a pair of well-known "double one" parameters, namely innodb_flush_log_at_trx_commit and sync_binlog . For security, the default value of these two parameters is 1. In order to quickly import the script, we can temporarily modify these two parameters. The following briefly introduces these two parameters:

innodb_flush_log_at_trx_commit：

The default value of innodb_flush_log_at_trx_commit is 1, which can be set to 0, 1, 2

If innodb_flush_log_at_trx_commit is set to 0, the log buffer will be written to the log file once per second, and the flush (to disk) operation of the log file will be performed at the same time. In this mode, when the transaction is committed, the operation of writing to disk will not be actively triggered.

If innodb_flush_log_at_trx_commit is set to 1, MySQL will write the log buffer data to the log file and flush (to disk) each time the transaction is committed.

If innodb_flush_log_at_trx_commit is set to 2, MySQL will write the log buffer data to the log file every time the transaction is committed. But the flush (flush to disk) operation will not be performed at the same time. In this mode, MySQL will perform a flush (to disk) operation every second.

sync_binlog：

The default value of sync_binlog is 1, which can be set to [0,N)

When sync_binlog = 0, MySQL will not synchronize to the disk but rely on the operating system to refresh the binary log, just like the mechanism of the operating system to refresh other files.

When sync_binlog =N (N>0), MySQL will use the fdatasync() function to synchronize its written binary log binary log to disk every N times.

These two parameters can be modified online. If you want to quickly import them, you can follow the steps below:

# 1.进入MySQL命令行 临时修改这两个参数
set global innodb_flush_log_at_trx_commit = 2;
set global sync_binlog = 2000;

# 2.执行SQL脚本导入
mysql -uroot -pxxxxxx testdb < testdb.sql

# 3.导入完成 再把参数改回来
set global innodb_flush_log_at_trx_commit = 1;
set global sync_binlog = 1;

        

        Another scenario is that your requirement is to create a new slave library, or you do not need to generate binlog. At this time, you can set not to record binlog temporarily when importing SQL scripts. You can add set sql_log_bin=0 at the beginning of the script; and then execute the import. This speeds up further. If binlog is not enabled on your MySQL instance, there is no need to execute this statement.

Summarize:

This article mainly introduces the method of quickly importing data, and there may be other methods to quickly import data, such as load data or write programs for multi-threaded insertion. The method described in this article is only suitable for manually importing SQL scripts. The following summarizes the methods mentioned in this article.

1. Use the command line tools that come with MySQL to export and import.
2. Using the extended insert method, one insert for multiple values.
3. Temporarily modify the innodb_flush_log_at_trx_commit and sync_binlog parameters.
4. Close binlog or temporarily not record binlog.

In fact, there are some other solutions, such as not creating an index first, inserting data and then adding an index; or first changing the table to a MyISAM or MEMORY engine, and then changing to an InnoDB engine after the import is complete.