Catch all kinds of locks of MySQL in one go
series of articles
4. The love-hate relationship between S lock and X lock "Deadly Kick MySQL Series 4"
5. How to choose ordinary index and unique index "Deadly Kick MySQL Series V"
Six or five minutes, let you understand how MySQL chooses indexes "Deadly Kick MySQL Series VI"
7. Strings can be indexed like this, do you know? "Deadly Kick MySQL Series Seven"
In the project, MySQL's error reports, exceptions, and long execution times are all sent to the DingTalk group, which is conducive to timely processing. Today we are going to talk about slow queries that cannot be reproduced.
1. Why there is "slow" SQL that cannot be reproduced
The redo log and binlog are explained in detail in the lifelong friend redo log and binlog "Deadly Kick MySQL Series II" . At this point, you know that when the transaction is committed during the update, instead of directly modifying the data in the database, the memory is updated first and the related operations are recorded in the redo log.
After all, it is necessary to flush the data in memory to the disk, which can also be called flushing dirty pages.
What are dirty pages and clean pages
Most sources mention dirty pages, so what exactly are dirty pages? Dirty pages When the data of the memory data page is inconsistent with the disk data, the memory page is called a dirty page.
When the memory page is written to the disk, the data pages of the memory and the disk are consistent, and the memory page is called a clean page at this time.
When do dirty pages become clean pages
The first
Innodb's redo log is full, that is, the write pos in the figure below catches up with the check point. At this time, all update operations of the system will stop.
Until the check point is advanced and the corresponding dirty pages are flushed to the disk, the redo log can continue to be written.
Under normal circumstances, the redo log log will not be full after the redo log is set according to the innodb_log_file_size parameter in the early stage of development.
the second
Due to insufficient memory, updating a statement will first update the memory and then update it to the redo log. If the memory is insufficient, new memory cannot be applied, and some data pages need to be eliminated. You need to flush dirty pages to disk.
Have you ever thought that since the update operation saves a copy of the memory and the redo log, can you directly eliminate the memory page, read the data page from the disk when there is a request, and then take out the redo log and apply it?
When the memory is full, the dirty pages are not flushed and they are eliminated directly. Then, the next time you request a clean page from the disk to the memory, you need to additionally judge whether the page has been modified in the redo log, and if so, you need to apply redo to it. log. This dirty page always needs to be flushed, but now there is a lack of additional operations to apply redo log. Therefore, the memory cannot be eliminated directly, but flushed directly when the memory is full.
In addition, the redo log is written cyclically. If you want to apply the redo log, the redo log must always exist and cannot be deleted. Violates the system design.
the third
MySQL brushes dirty pages during system low-peak periods
the fourth
When MySQL is shut down normally, all dirty pages in memory will be flushed to the disk. After restarting, the data will be read directly from the disk, and the startup speed will be very fast.
in conclusion
At this point, you should understand that the inexplicably slow SQL is caused by flush, so how do these four situations affect MySQL?
2. The impact of the four types of flush on performance
The third and fourth cases will not cause slow execution of MySQL due to flushing. One is the idle period of the system, and the other is that the database is about to be closed.
The redo log is full and needs to flush dirty pages
This situation has been given a plan in the second article. Once the redo log is filled with the entire system, it will no longer accept update operations, and all update operations must be stagnant until the check point is advanced.
expand
The innodb_log_file_size parameter is provided in MySQL to optimize the redo log log.
There are also some calculation rules for the setting of innodb_log_file_size, which will be introduced to you below.
If the innodb_log_file_size setting is too small, the redo log file will be switched frequently, and the checkpoint of the database will be triggered frequently, resulting in an increase in the number of records updated to the data file, thus affecting the IO performance.
Similarly, if there is a large transaction, and all the redo log logs are full, but not completed, the log cannot be switched, which will cause MySQL to block directly.
The innodb_log_file_size setting is too large. Although the IO performance is greatly improved, when MySQL restarts or crashes, the recovery time will be prolonged because the redo log file is too large. And this recovery time is often out of control.
How to set innodb_log_file_size reasonably?
Use a script to execute regularly, record the sequence number of the corresponding time and then take the average value, the calculated error will be minimized. The sequennumber is when each binlog is generated, the value starts from 1, and then increases. Every time a transaction is added, the sequennumber is increased by 1.
Insufficient system memory, need to brush dirty pages
The memory management in Innodb is the buffer pool. The memory page can be seen in the above three states, unused, used is a clean page, used is a dirty page.
For a long-running library, there are very few unused pages, and when memory is insufficient, only the oldest unused data pages can be eliminated from memory.
If a clean page is eliminated, it is directly released for use, but if it is a dirty page, the dirty page must be flushed first and turned into a clean page for reuse.
If the queried data is not in memory, the data needs to be read from the disk. If too much data is read, multiple dirty pages need to be eliminated, which will result in a long query time.
The redo log is full, and all update systems are not executed, which is unacceptable for most businesses.
In order to prevent this from happening, it is necessary to control the frequency of brushing dirty pages.
3. How to set the speed of brushing dirty pages
The speed of brushing dirty pages to disk must be related to the IO capability of the system. In MySQL, innodb_io_capacity controls the speed of brushing dirty pages.
When flushing dirty pages from the buffer (check point), the number of dirty pages flushed per second is equal to the value of innodb_io_capacity.
This value can be set to the IOPS of the disk, which can be tested using the fio tool. The specific use will not be discussed here.
The speed of brushing dirty pages is also determined by the proportion of dirty pages and the speed of redo log writing to the disk.
The parameter innodb_max_dirty_pages_pct is the upper limit of the dirty page ratio. In MySQL 8.0, this ratio defaults to 90%, and MySQL 5.6 is still 75%.
In general, the value of innodb_io_capacity is set to the upper limit of the dirty page ratio and the log sequence number when writing the redo log log minus the value of checkpoint, whichever is the largest.
The calculation formula of the dirty page ratio is Innodb_buffer_pool_pages_dirty/Innodb_buffer_pool_pages_total, the specific execution command is
mysql> select VARIABLE_VALUE into @a from global_status where VARIABLE_NAME = 'Innodb_buffer_pool_pages_dirty';
select VARIABLE_VALUE into @b from global_status where VARIABLE_NAME = 'Innodb_buffer_pool_pages_total';
select @a/@b;
In this SQL statement, you can see that the global_status table is used in the performance_schema library. Use performance_schema needs to be executed before executing the command.
When your MySQL write speed is very slow, TPS is very low, and IO pressure is not high, you need to check the place
When this problem occurs, consider whether the parameter value of innodb_io_capacity is reasonable.
The default value of the server with 1 core and 2G is 200, and it is 2000 on the company server, which is also related to the server configuration.
Fourth, interesting parameters
The default value of the parameter innodb_flush_neighbors in MySQL 8.0 is 0.
When a query needs to flush a dirty page during execution, if the data page next to the data page happens to be a dirty page, the data page will be flushed together, and the associated logic will continue. Will make SQL queries slower.
Persistence in learning, perseverance in writing, perseverance in sharing are the beliefs that Kaka has upheld since her career. I hope the article can bring you a little help on the huge Internet, I am Kaka, see you in the next issue.