Verbatim large column https://www.dazhuanlan.com/2019/08/26/5d634d54a761b/
InnoDB is a disk-based storage, and which records are managed in a manner page.
In the database system, because the gap between the speed of the CPU and disk speed, disk-based database systems typically by pool technology to improve the overall performance of the database. Pool is simply a memory area by the memory speed to compensate for the slower disk speed impact on database performance.
Read operation performed in the database page, the first page read from the disk is stored in the buffer pool, a process known as the "FIX" page in the buffer pool. The next time you read the same page, is limited to determine whether the page in the buffer pool, if the buffer pool, called the page is in the buffer pool hit directly read the page. Otherwise, read pages on disk.
Locality principle : When a data is used, the data in its vicinity is also often used immediately
Disk read-ahead : the access speed of the disk is often one hundred sub-divided main memory, so in order to improve efficiency, to minimize the disk I / O. For this purpose, the disk read demand is often not critical, but each will preread, even if it requires only one byte, the disk will begin from this position, a length of data sequentially read back into the memory
InnoDB storage engine when processing the client's request, when the need to access data in a page, it will put the complete data of all the pages loaded into memory, which means that even if we only need to access a page of a record, that's You need to first load the data for the entire page into memory.
Modification operations on the database page, the first page in the buffer pool modify, and then at a certain frequency flushed to disk, It should be noted that the pages are flushed to disk operation from the buffer pool is not in every page It triggered when changes occur, but through a mechanism called CheckPoint refresh back to disk, the same which is to improve the overall performance of the database.
In summary, the size of the buffer pool directly affects the overall performance of the database. InnoDB buffer pool configuration parameters can beinnodb_buffer_pool_size来设置
mysql> show variables like 'innodb_buffer_pool_size'G
*************************** 1. row ***************************
Variable_name: innodb_buffer_pool_size
Value: 134217728
1 row in set (0.01 sec)
Pool of the cache page data types: index page, data pages, pages Use the undo, insert buffer (insert buffer), the adaptive hash index (adaptive hash index), the lock information stored InnoDB (lock info), data dictionary information (data dirctionary) and the like. Data pages and index pages only accounts for a large part of the buffer pool only
InnoDB buffer pool allows multiple instances, each page is evaluated based on a hash to different fragments of the buffer pool instances, the benefits of doing so is to reduce competition for resources inside the database, increasing the concurrent processing of the database, you can innodb_buffer_pool_instancesbe configured
mysql> show variables like 'innodb_buffer_pool_instances'G
*************************** 1. row ***************************
Variable_name: innodb_buffer_pool_instances
Value: 1
1 row in set (0.00 sec)
You can also use show engine innodb statusGthe command to observe the buffer pool
Buffer Pool default page cache size on the disk and the default page size is the same, are 16KB. In order to better manage these cached pages in the Buffer Pool, InnoDB buffer for each page creates some control information, the control information includes the number of the page table space belongs, page number, cache page address in the Buffer Pool , list node information, some of the information and lock LSN information, etc.
Each control information corresponding to the page cache memory size is the same, which is placed in the front part of the Buffer Pool, it is stored in the cache page back Buffer Pool.
Buffer pool cache data comprises Page Cache, Change Buffer, Data Dictionary Cache, usually 80% of the MySQL server will be allocated to the physical memory Buffer Pool
InnoDB memory structure including Buffer Pool, Change Buffer, Adaptive Hash Index and Log Buffer of four parts. If memory from the point of view, Change Buffer and Adaptive Hash Index memory occupied belong Buffer Pool, Log Buffer memory occupied by independent and Buffer Pool
LRU List、Free List和Flush List
Free list for which the record buffer pool cache page are free.
Flush the list used to record which cache page buffer pool is dirty page
Dirty page if we modify the data in the buffer pool cache of a page, and that page on disk and it is inconsistent, so the cache page is also known as dirty pages (dirty page). If a modification occurs immediately synchronized to the performance it will seriously affect the process of the corresponding page on the disk. So it is necessary to record a list which cache pages need to be flushed to disk
LRU list buffer pool size is limited, can not grow indefinitely, if the free list has no extra free buffer page, you need to put some old cached page removed from the buffer pool, then add the new page. When the pool is no longer free cached pages, we need to eliminate part of the cache of recently used page. LRU list is used in accordance with the principle of least recently used to eliminate caching pages, when we need to access a page, put the page buffer adjusted to the head of the LRU list, so that the tail of LRU list is the least recently used cached pages
Pre-reading
There InnoDB in optimizing the I / O is one of the more important characteristics of the prefetch, a prefetch request is i / o requests, it will asynchronously buffer pool previously fetched plurality of pages, these pages are expected soon need these requests all incorporated within a range of pages.
Database when the requested data, will be read request to the file system, into the request queue; removed from the associated process requests a read request in the queue, to the associated data area (memory, disk) reading data according to the needs; extracted data , into the response queue, the last data in the database will be removed from the response queue, a data read operation to complete the process. The process then continues processing the request queue (if the database is full table scan, the data read request will occupy request queue), it is determined whether the data following the data read request several adjacent, then itself according to the processing amount IO bandwidth system for pre-read, read requests are merged, one-time data into a plurality of read response queue, then the database is removed. (Thus, a physical read operation, multi-page data read)
InnoDB two prefetch algorithms to improve I / O performance: linear prefetch (linear read-ahead) , and random read-ahead (randomread-Ahead) , focus on the linear pre-read area to read the next advance into the buffer pool, the random read-ahead will focus on the current region to the rest of the page read buffer pool in advance.
Linear pre-reading
innodb parameter innodb_read_ahead_thresholdtrigger time innodb read ahead operation control. If a region of the page is sequentially read parameter exceeds or is equal to the variable, asynchronous Innodb will read the next buffer pool in the region, innodb_read_ahead_thresholdcan be set to any value of 0-64, the default value is 56, the higher the value, the more stringent checks the access mode.
mysql> show variables like 'innodb_read_ahead_threshold';
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| innodb_read_ahead_threshold | 56 |
+-----------------------------+-------+
1 row in set (0.03 sec)
InnoDB only when access to the current page area 56 in order to trigger the linear pre-read request to read a next memory area. In the absence of this variable, when access to the last page of the area, InnoDB will decide whether the next area to put in the buffer pool.
Random read-ahead
When the same area of some of the pages found in the buffer pool, InnoDB will be in the area of the remaining pages in the buffer pool read together. InnoDB parameter innodb_random_read_aheadcontrolling the random prefetching opening.
mysql> show variables like 'innodb_random_read_ahead';
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| innodb_random_read_ahead | OFF |
+--------------------------+-------+
1 row in set (0.04 sec)
InnoDB monitor pre-reading
By show engine innodb status;monitoring the pre-read information
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
By Innodb_buffer_pool_read_aheadand Innodb_buffer_pool_read_ahead_evictedevaluate the effectiveness of pre-reading algorithm
mysql> show global status like '%read_ahead%';
+---------------------------------------+--------+
| Variable_name | Value |
+---------------------------------------+--------+
| Innodb_buffer_pool_read_ahead_rnd | 0 |
| Innodb_buffer_pool_read_ahead | 352999 |
| Innodb_buffer_pool_read_ahead_evicted | 0 |
+---------------------------------------+--------+
3 rows in set (0.07 sec)
MySQL buffer pool contamination
When one SQL statement to batch scan large amounts of data, it may lead to all the pages are swapped out of the buffer pool, resulting in a large number of hot data to be swapped out, a sharp decline in the performance of MySQL, this is called the buffer pool contamination.
If a very large number of records in the table, then the table will occupy a particularly large number of pages, when performing a full table scan of the table will be a large number of pages loaded into the Buffer Pool, which also means that the Buffer Pool All pages are replaced once, and then the other query was executed once loaded from disk into the Buffer Pool operations in the implementation of Shiyou. The frequency of such statements to perform a full table scan is not high, each execution should replace the cached page of a Buffer Pool, which seriously affect the use of other inquiries relating to Buffer Pool, which greatly reduces the cache hit rate .
LRU algorithm
According to the previous description can know for certain SQL operations may cause the page buffer is flushed out of the pool, thus affecting the efficiency of the buffer pool. For example, as an index or data scanning operation, these operations need access to many pages in the table, even all the pages, and these pages are typically only required for operation in the query is not active hot data. If the page when the header is placed in LRU list, then very hot might need to remove the data page from the LRU linked list, and the next page to be read, InnoDB need to access the disk again.
Therefore, the LRU algorithm for InnoDB buffer pool made improvements LRU list added to a midpoint location. The new pages to read, although the latest access pages, but not directly into the LUR to the head of the list, but put midpoint position to the LRU list.
In innodb to the list after the midpoint called old list before the list is called the new list, you can simply understood as a new page in the list are the most active hot data.
By innodb_old_blocks_pctcontrolling the distribution of the new and old, the default value is 37
mysql> show variables like 'innodb_old_blocks_pct'G
*************************** 1. row ***************************
Variable_name: innodb_old_blocks_pct
Value: 37
1 row in set (0.01 sec)
After the LRU list into old and new, InnoDB logic processing according to the following
- When a page on disk is loaded into the Buffer Pool for the first time in a cache page, the page cache will be put old head of the list. This will be gradually expelled for pre-read Buffer Pool did not follow-up visit a page from the old list, without affecting the young are more frequently used list of cached pages.
- Recorded at the time of a cache page in the old list is accessed for the first time in its corresponding control block down the access time, if the time of a subsequent access time with the first access within a certain time interval, then the page will not be moved from the old to the new head of the list, or move it to the new list of the head of the list
Because full table scan has a feature that its execution frequency is very low, and in the course of a full table scan, even if a page has many records, that is, to many visits it takes time this page is Extremely few.
InnoDB by innodb_old_blocks_timethe mid to position control page read parameter is how long to wait before added to the hot end of the LRU list. The default value of 1000 milliseconds
mysql> show variables like 'innodb_old_blocks_time'G
*************************** 1. row ***************************
Variable_name: innodb_old_blocks_time
Value: 1000
1 row in set (0.00 sec)
LRU list with the page management have been read, but when the database has just started, LRU list is empty, ie without any page, then the pages are stored in the Free list. When the required page from the buffer pool, first from the free list for available free pages, the page will be removed if put into the LRU list from the free list, or according to the LRU algorithm, eliminating the LRU end of the list page, the memory page to the new slice. When the old page from the LRU portion of the new part, the operation occurring at that time is called page made young, and because innodb_old_blocks_timethe leads provided per page called page not made young from old to move the operating portion of the new section.
In Buffer Pool inside the database, whether it is old data new list or if the list is not access to, the final will be moved to the end of the list as the page being eliminated.
show engine innodb status
You can use show engine innodb statusGa command to observe the state of usage and operation of the LRU list and free list
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137428992
Dictionary memory allocated 3807133
Buffer pool size 8192
Free buffers 1024
Database pages 7141
Old database pages 2616
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 127464, not young 10579743
0.00 youngs/s, 0.00 non-youngs/s
Pages read 393131, created 205050, written 19241821
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 7141, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
Total large memory allocated 137428992Innodb allocated memory size
Dictionary memory allocated 3807133Innodb assigned to the data dictionary memory size
Buffer pool size 8192 innodb_buffer_pool size (page)
Free buffers 1024 The number of free pages innodb_buffer_pool lru list
Database pages 7141 Non-free number of pages innodb_buffer_pool lru list
Old database pages 2616 The number of pages innodb_buffer_pool sub-list of old
Modified db pages 0 The number of dirty pages in innodb_buffer_pool
Pending readsAnd Pending writesdisplay of the pending reads and writes
Buffer pool hit rate 1000 / 1000...Shows the buffer pool hit rate Innodb, usually to ensure that more than 998/1000. If not, consider increasing the buffer pool size, and optimize your queries
Pages read ahead 0.00/sIt shows the number of second linear pre-read (read).
evicted without accessIt shows pages read per second
Random read ahead 0.00/sIt shows the number of random read-ahead per second.
Buffer pool size A total of 8191 pages, a total of 8191 * 16K buffer pool space, Free buffers represents the number of the current page Free list. Database pages indicates the number of pages in the LRU list. May Free buffers + Database pages may not equal Buffer pool size, since the page buffer pool may also be assigned to the insert buffer, adaptive hash index, the lock information (lock info), data dictionary information,
buffer pool configuration
- innodb_buffer_pool_size: This value is the total size of the set of InnoDB Buffer Pool;
- innodb_buffer_pool_chunk_size: size of chunk size of the execution unit of InnoDB Buffer Pool. This is inside a relationship to determine what is best in accordance with this setting
innodb_buffer_pool_size=innodb_buffer_pool_chunk_size * innodb_buffer_pool_instances*N（N>=1）; - innodb_buffer_pool_instances: InnoDB Buffer Pool set the number of instances, each instance has its own separate list management Buffer Pool;
- innodb_old_blocks_pct: default InnoDB Buffer Pool midpoint position, the default value is 37, maximum 100, which is what we called the 3/8 position, you can set up their own;
- innodb_old_blocks_time: Set retention time data stored in the Buffer Pool inserted inside list is not changed when the position of the time;
- innodb_read_ahead_threshold: MySQL parameter control when to prefetch, MySQL may be controlled to pre-read data when the sensitivity of the data, if the data pages Buffer Pool frequent value greater than the value stored inside the innodb_read_ahead_threshold, InnoDB starts an asynchronous prefetch operating;
- innodb_random_read_ahead: Disabled default, the parameter controlling the read-ahead, it will open to read, but do not use the linear pre-random read-ahead;
- innodb_adaptive_flushing: Specifies whether the dynamic adaptive refresh dirty pages to disk, this is based on MySQL load their own decisions. But still try not to set up, let MySQL to manage their own;
- innodb_adaptive_flushing_lwm: Close adaptive_flushing words will be useful for line marking the lowest percentage of redo log usage, when reached this value, it will refresh the dirty pages, the default is 10;
- innodb_flush_neighbors: control whether to refresh the dirty data pages Buffer Pool dirty when dirty data pages in the same area together to refresh, the default value of 1;
- innodb_flushing_avg_loops: save for the InnoDB InnoDB rinse several times a snapshot of the state of the number of iterations before Buffer Pool, the default value is 30, increase it, rinse it becomes slow. Then rinsing the reduced frequency will become high;
- innodb_lru_scan_depth: a control parameter LRU algorithm is used to control the background process page_cleaner Buffer Pool Refresh location dirty pages;
- innodb_max_dirty_pages_pct: InnoDB Buffer Pool parameters will refresh the data without letting the percentage of dirty data exceeds this value;
- innodb_max_dirty_pages_pct_lwm: InnoDB will automatically maintain a background job automatically clears the dirty data from Buffer Pool which, when dirty pages Buffer Pool in the occupancy ratio of time to reach the set value innodb_max_dirty_pages_pct_lwm, it will automatically clear the dirty pages Buffer Pool;
- innodb_buffer_pool_filename: Specify the document name;
- innodb_buffer_pool_dump_at_shutdown: whether to keep the current configuration of the InnoDB buffer pool state to avoid after the server is restarted, but also through a long warm-up time;
- innodb_buffer_pool_load_at_startup: This parameter specifies the start, after the database will automatically restart the warm-up, read the information stored before the restart of the Buffer Pool;
- innodb_buffer_pool_dump_now and innodb_buffer_pool_load_now when the database has been brought to, we forget that previously specified, you can specify immediately restored;
- innodb_buffer_pool_dump_pct: How much data set about the recovery in the Buffer Pool;
- innodb_buffer_pool_load_abort: termination Buffer Pool restore, you can specify load operation.
Reference material
"MySQL InnoDB storage engine technology insider 2nd Edition"
"MySQL is how it works: Understanding MySQL from the stalk."
https://www.cnblogs.com/geaozhang/p/7397699.html
https://mp.weixin.qq.com/s/ZPXcsogmO9BkKMKNLQxNLA
https://mp.weixin.qq.com/s/nA6UHBh87U774vu4VvGhyw