The adjustment record of the insertion speed and reading speed of mysql database of tens of millions (transfer)

Under normal circumstances, there is no problem in mysql reading and inserting millions of data, but when it reaches tens of millions, it will be very slow. Let's take a look at the adjustment records of the insertion speed and reading speed of mysql database of tens of millions.
(1) Improve the database insertion performance The central idea: try to write data to the Data File at one time and reduce the checkpoint operation of the database. This time, the following four configuration items have been modified:
1) Set the innodb_flush_log_at_trx_commit configuration to 0; according to past experience, set it to 0, and the insertion speed will be greatly improved.
0: Write the log buffer to the log file and flush the log file every second, but do nothing at transaction commit.
1: the log buffer is written out to the log file at each transaction commit and the flush to disk operation is performed on the log file
2: the log buffer is written out to the file at each commit, but the flush to disk operation is not performed on it
2) Adjust the innodb_autoextend_increment configuration from the default 8M to 128M
. This configuration item is mainly used when the tablespace space has been After it is full, how much space needs to be automatically expanded by the MySQL system. Each time the tablespace is expanded, each SQL will be in a waiting state. Increasing the auto-expansion Size can reduce the number of tablespace auto-expansion times.
3) Adjust the innodb_log_buffer_size configuration from the default 1M to 16M
. This configuration item is used to set the innodb database engine write log buffer area; increasing this buffer segment can reduce the number of times the database writes data files.
4) The innodb_log_file_size configuration is adjusted from the default 8M to 128M
. This configuration item is used to set the size of the UNDO log of the innodb database engine; thus reducing the database checkpoint operation.
After the above adjustment, the insertion speed of the system has been increased to about 1W per second due to the original tens of thousands in 10 minutes; Note: The above parameter adjustment needs to be adjusted according to different machines. In particular, innodb_flush_log_at_trx_commit, innodb_log_buffer_size, and innodb_log_file_size need to be adjusted carefully; it involves the disaster recovery of MySQL itself.
(2) Improve the reading speed of the database, and the reading speed at the database level is mainly due to several reasons: simplifying SQL, adding indexes and partitioning; After checking the program, SQL is already the simplest, and indexes have been added to the query conditions. We can only use the weapon: table partitioning.
Database MySQL partition preparation: In MySQL, a tablespace is a data file that stores data and indexes.
Modify the S11 database to support multiple tablespaces due to the shared tablespace; modify
the wb_user_info_sina and wb_user_info_tx tables to their own independent tablespaces; (Sina: 1700W data, 2.6G big data file, Tencent 1400W, 2.3G big data file);
Partition operation:
delete the existing primary key and index first, and
recreate the joint primary key of id and uid
Then use uid as the key value to partition. At this time, go to /var/data/mysql to view the data files, and you can see that the independent table spaces of the two large tables have been divided into several smaller independent partition spaces. (At this time, if the uid is used as the retrieval condition for query, the speed will not be improved; because the key value is only the partition of the data storage, and no partition index will be established. I am very depressed that this is not a little bit worse than Oracle.)
Then use the uid Index on the field. Go to the /var/data/mysql folder to view the data files again, and I am very depressed to find that the size of each partition is actually large. MySQL still stores indexes and data in the same tablespace as usual. If the index and data can be separated, it can be better managed.
After the above adjustments, the system reading speed has not been improved for the time being; basically, the 5K data update is completed in 2~3 seconds.
Supplementary information for MySQL database insertion speed adjustment:
MySQL has increased the insertion speed from 1000 entries/minute to 10000 entries/second from the very beginning. I believe that everyone has been waiting for the relevant introduction. Below is the whole process of my tuning. The central idea of ​​improving database insertion performance:
1. Try to make the database write to Data File
at one time. 2. Reduce the checkpoint operation of the database.
3. Try to buffer data in the program, and perform batch insertion and submission
.
Content, as an amateur DBA made the following adjustments to the MySQL service:
Modify the configuration of the MySQL server responsible for recording and recording, and improve the overall write speed of MySQL; the specific values ​​are the following three database variables: innodb_autoextend_increment, innodb_log_buffer_size, innodb_log_file_size; the default values ​​of these three variables are 5M, 8M, and 8M, respectively, according to the server memory size and specific use. According to the situation, the three were modified to: 128M, 16M, 128M. At the same time, the original 2 Log Files are also changed to 8 Log Files. This modification mainly satisfies the first and second points, such as: adding innodb_autoextend_increment is to avoid the checkpoint operation of MySQL due to frequent automatic expansion of Data File;
change the large table to an independent table empty and partition it, and then hang different partitions in several different hard disk arrays.
After completing the above modification operations; I see the following happy results:
Get test results:
Query OK, 2500000 rows affected (4 min 4.85 sec)
Records: 2500000 Duplicates: 0 Warnings: 0
Query OK, 2500000 rows affected (4 min 58.89 sec) )
Records: 2500000 Duplicates: 0 Warnings: 0
Query OK, 2500000 rows affected (5 min 25.91 sec)
Records: 2500000 Duplicates: 0 Warnings: 0
Query OK, 2500000 rows affected (5 min 22.32 sec)
Records: 2500000 Duplicates: 0 Warnings: 0
The amount of data in the last table:
+------------+
| count(*) |
+------------+
| 10000000|
+------------+
From the above results, the increase in the amount of data will have a certain impact on the insertion performance. However, the overall speed is still very negotiable. In less than a day, 400 million data can be processed normally. It is expected that the database bottleneck has been solved ingeniously, and the result is that the program "ape" complained to me bitterly.