I. Introduction

Previous article already mentioned ( https://blog.csdn.net/qq_41453285/article/details/104083744 ), designed to buffer pool in order to coordinate CPU speed and disk speed gap. Therefore, operation of the first page in the buffer pool are completed . If a DML statement, such as changing the Update or Delete records page, so this time the page is dirty . That is the new version of the disk than the page buffer pool. The new version of the database requires page refresh from the buffer pool to disk

ACID properties of D (persistence) of

If every time a page is changed, the new version will refresh the page to disk, then this overhead is very large . If the hot data set in a few pages, then the performance of the database will become very poor. Meanwhile, if the downtime occurs when a new version of the page from the buffer pool will be flushed to disk, data can not be restored

In order to avoid data loss problems occur , the current transaction database systems generally have adopted the Write Ahead Log strategy, that when the transaction commits, the first to write redo log, then modify the page . When downtime occurs due to the loss of data caused by redo logs to complete the recovery of data . That requires ACID transactions of D

Second, why Checkpoint technology design

Consider the following scenario, if the redo log can be infinitely increased, while the pool is large enough to buffer data from all databases , it is not necessary to the new version of the page to refresh the buffer pool back to disk. Because when downtime occurs, the data can be recovered through the entire database system downtime redo logs to the time of the occurrence, but requires two preconditions:

1. All the data buffer pool can cache database

For the first prerequisite, experienced users know when the database is just the beginning of creation, there is no data in the table. Buffer pool cache can indeed all database files. However, with the promotion of the market, the increase of users, the products are more and more attention, use is also growing. At this time in charge of the back-end storage capacity of the database will certainly continue to grow. The current 3TB of MySQL has not uncommon, but it is very rare 3TB of memory

2. redo logs can increase wireless

Corresponding to the second premise conditions: redo log can be increased infinitely. Maybe it is, but it requires cost is too high, while not easy to operation and maintenance. SA or DBA does not know whether and when the redo logs has been close to the threshold of disk space available, and to make storage devices support the dynamic expansion course also requires a certain skill and device support

Even if the above two conditions are met, then there is a case to be considered: the recovery time after the database downtime. When even a few years when the database is running for a few months, when downtime occurs, reapply the redo logs will be very long time, this time the code is also very large recovery

The object of Checkpoint (Checkpoint) technology is to address the following questions:
- Shorten the recovery time of the database
- When the pool is not enough, the dirty pages are flushed to disk
- When redo logs are not available, flushing dirty pages
When the database downtime occurs, the database does not need to redo all the log, because the previous Checkpoint pages have been refreshed back to disk. Therefore, the database only need to redo logs after Checkpoint recovery. This greatly reduces recovery time
In addition, when the pool is not available, according to the LRU algorithm will overflow the least recently used page, if this page is dirty pages, then you need to enforce Checkpoint, dirty pages is a new version of the page to disk brush
Redo log unavailable occur because the current transaction database system designed for redo logs are recycled, are not allowed to grow indefinitely, which from a cost and management are more suffering. Redo logs may be reusable parts mean that the redo log are no longer needed, i.e., when the database downtime occurs, the recovery operation does not need to redo log database this portion, thus this part may be covered reused. If the redo logs at this time also need to use, you must force a Checkpoint, the buffer pool to refresh the page at least to the current location of the redo log

Third, the log sequence number LSN

For the purposes of InnoDB, which is marked by LSN version. LSN is the 8-byte number, the unit is a byte.
Each page has LSN, redo log also LSN, Checkpoint also LSN. Can be viewed with the following command to display the information:

show engine innodb status\G;

Four, Sharp Checkpoint and Fuzzy Checkpoint

In the InnoDB storage engine, Checkpoint choose the time, the conditions and the occurrence of dirty pages and so very complex . The Checkpoint much of what is nothing less than to brush dirty pages in the buffer pool back to disk. The difference is that each time the number of disk pages to refresh, take a time where dirty pages, and what time-triggered Checkpoint
Inside the InnoDB storage engine, there are two Checkpoint, namely:
- Sharp Checkpoint
- Fuzzy Checkpoint

Sharp Checkpoint

Sharp Checkpoint occur when the database is shut down will refresh all the dirty pages to disk , which is the default mode of operation, ie parameter innodb_fast_shutdown = 1

Fuzzy Checkpoint

If the database also uses Sharp Checkpoint at run time, then the availability of the database will be greatly affected. So InnoDB or use the Fuzzy Checkpoint refresh the page, that is only part of the dirty page refresh , rather than flush all dirty pages to disk

Fuzzy Checkpoint will happen in the following situations:

1.Master Thread Checkpoint

2.FLUSH_LRU_LIST Checkpoint

3.Async/Sync Flush Checkpoint

4.Dirty Page too much Checkpoint

①Master Thread Checkpoint

For Master Thread (later article will detail) Checkpoint occurred in almost at a rate per second refresh every ten seconds or a certain percentage of the page to disk from the list of dirty pages in the buffer pool . This process is asynchronous, that at this time the InnoDB storage engine can perform other operations, the thread does not block the user's query

②FLUSH_LRU_LIST Checkpoint

working principle:

FLUSH_LRU_LIST Checkpoint is because InnoDB storage engine needs to ensure LRU list needs to have almost 100 free pages available . If there is no available free page 100, then InnoDB will be the end of the LRU list of pages removed , if these pages have dirty pages, you need to be Checkpoint

These pages from the LRU list, so called FLUSH_LRU_LIST Checkpoint

Before InnoDB 1.1.x version, you need to check whether there is enough free space operations LRU list occurs in the user query thread, obviously can block a user's query operation

From MySQL 5.6 version, which is InnoDB 1.2.x version, this check is placed in a separate thread Page Cleaner carried out , and the user can innodb_lru_scan_depth parameter to control the number of available pages of LRU list , the value defaults to 1024
show variables like 'innodb_lru_scan_depth'\G;
③Async/Sync Flush Checkpoint

Async / Sync Flush Checkpoint refers to the redo log files are not available in the case, then the need to force some of the pages are flushed to disk , at a time when the dirty pages are selected from the list of dirty pages

If that has been written to the redo log of LSH is the redo_lsn, will have to refresh the page to disk latest LSN recorded as checkpoint_lsn, can be defined:

And then define the following variables:

If each redo log file size is 1GB, and defines two redo log files, the total size of the redo log files to 2GB. So async_water_mark = 1.5GB, sync_water_mark = 1.8GB, then:

When checkpoint_age <async_water_mark: you do not need to flush any dirty pages to disk

When async_water_mark <checkpoint_age <sync_water_mark: After trigger Async Flush, refreshing enough dirty pages to disk Flush from the list so refreshing to meet checkpoint_age <async_water_mark

checkpoint_age> sync_water_mark case rarely occurs in general, unless the redo log file set is too small, and performing the LOAD DATA BULK INSERT operation is similar. After this time trigger Sync Flush operation, refresh enough dirty pages to disk Flush from the list so refreshing to meet checkpoint_age <async_water_mark

Thus, Async / Sync the Flush Checkpoint is to ensure the availability of recycled redo log

Before InnoDB 1.2.x version, Async blocking the Flush Checkpoint will find the user query threading issues, and Sync Flush Checkpoint blocks all user queries thread, and wait for the dirty page refresh is complete. InnoDB1.2.x version from the beginning , this part of the refresh operation is also put into separate Page Cleaner Thread in, it does not block the user's query thread

The official version of MySQL and can not view or refresh the page is carried Checkpoint from the LRU list Flush from the list, do not know the number of Async / Sync Flush because redo logs generated, but InnoSQL version provides a method, by the following command to observe:
show engine innodb status\G;
④Dirty Page too much Checkpoint

Works: Dirty Page TOO much, that is, too many dirty pages, resulting in InnoDB storage engine to force Checkpoint. Its purpose is generally available in the pool in order to ensure sufficient buffer page

Which may be controlled by the parameter innodb_max_dirty_pages_pct:

75 represents a value shown above, when the number of buffer pool dirty pages to occupy 75% force a Checkpoint , a portion of the refresh dirty pages to disk. Before InnoDB 1.0.x versions, the parameter defaults to 90, after the versions are 75
show variables like 'innodb_max_dirty_pages_pct'\G;