Article directory
Change Buffer
When a data page needs to be updated, if the data page is in memory, it will be updated directly, and if the data page is not in memory, InnoDB will cache these update operations in the change buffer without affecting data consistency , so that this data page does not need to be read from disk.
When the next query needs to access this data page, read the data page into the memory, and then execute the operations related to this page in the change buffer. In this way, the correctness of the data logic can be guaranteed.
It should be noted that although the name is called change buffer, it is actually persistent data. In other words, the change buffer is copied in memory and written to disk.
The process of applying the operations in the change buffer to the original data page to get the latest result is called merge.
In addition to accessing this data page will trigger the merge, the system has a background thread that will merge periodically. The merge operation is also performed during a normal database shutdown.
Obviously, if the update operation can be recorded in the change buffer first to reduce disk reads, the execution speed of the statement will be significantly improved.
Moreover, reading data into memory needs to occupy the buffer pool, so this method can also avoid occupying memory and improve memory utilization.
So, under what conditions can the change buffer be used?
For a unique index , all update operations must first determine whether the operation violates the unique constraint.
For example, to insert the record of unique_key = 4, it is necessary to first determine whether the record of unique_key = 4 already exists in the table, and this must be determined by reading the data page into memory. If it has already been read into the memory, it will be faster to update the memory directly, and there is no need to use the change buffer.
Therefore, the update of the unique index cannot use the change buffer, and in fact only ordinary indexes can be used. The change buffer uses the memory in the buffer pool, so it cannot be increased infinitely. The size of the change buffer can be dynamically set by the parameter innodb_change_buffer_max_size. When this parameter is set to 50, it means that the size of the change buffer can only occupy 50% of the buffer pool at most.
Therefore, for the case where the data itself exists in memory, the performance gap between ordinary and unique indexes is not that big.
However, if the data does not exist in the memory, the unique index cannot use the change buffer because it needs to read the data from the disk into the memory, so the performance is much worse.
Through the above analysis, we have made it clear that the use of change buffer can accelerate the update process, and it is also clear that change buffer is only used in ordinary index scenarios, not applicable to unique indexes.
So, now there is a question:
Can the change buffer be used to speed up all scenarios of ordinary indexes?
Because the merge is the moment when the data is actually updated, and the main purpose of the change buffer is to cache the record change actions, so before a data page is merged, the more changes recorded in the change buffer (that is, the number of changes on this page) The more times you update), the greater the benefit.
Therefore, for businesses that write more and read less, the probability of the page being accessed immediately after writing is relatively small, and the use of change buffer is the best at this time. This kind of business model is common for billing and log systems. Conversely, assuming that the update mode of a business is to query immediately after writing, even if the conditions are met, the update will be recorded in the change buffer first, but the merge process will be triggered immediately because the data page will be accessed soon. In this way, the number of random access IOs will not be reduced, but the maintenance cost of the change buffer will be increased. Therefore, for this business model, the change buffer has a side effect.
Buffer Pool
Buffer Pool in MySQL is a mechanism introduced to optimize disk I/O. The page size inside MySQL and the page size of the operating system can be different. The page size in MySQL is usually fixed, and the default is 16KB.
Buffer Pool is a memory area used to cache data pages read and maintained from disk. Its function is to reduce disk I/O operations and improve database performance and response speed. When querying or modifying data, MySQL first checks whether the required data page exists in the buffer pool. If it exists, it can directly read or write data from memory, thus avoiding frequent disk read and write operations.
The size of the buffer pool can be set by configuring the parameter innodb_buffer_pool_size. It is very important to set the size of the buffer pool reasonably. A buffer pool that is too small may cause frequent disk I/O, while a buffer pool that is too large may waste precious memory resources.
In order to make better use of the buffer pool, the following aspects can be considered:
Set an appropriate buffer pool size according to the memory size of the system and the access mode of the database. This requires comprehensive consideration of the memory requirements of other processes and services in the system.
Monitor the usage of the buffer pool, for example, use the show global status like 'Innodb_buffer_pool%' command to check the buffer pool hit rate, usage rate and other indicators. The size of the buffer pool can be adjusted according to the monitoring results.
Make sure your table's indexes are optimized enough to minimize disk access.
For how to ensure the data consistency between the data in the Buffer Pool and the disk, the InnoDB engine adopts the Write-Ahead Logging mechanism. Before modifying the data, InnoDB will first write the modified data to the log file (Redo Log), and then update the data page to the Buffer Pool. In this way, even if the system crashes or restarts, the data consistency can be guaranteed through the recovery operation of the log.
(WAL) write-ahead logging is a common database transaction processing strategy designed to ensure data durability and consistency. In MySQL, when performing data modification operations, the InnoDB storage engine first writes the modification operations to log files (called redo logs or write-ahead logs), and then updates the corresponding data pages to the Buffer Pool. The advantage of this is that even if a system crash or power failure occurs before the update to the disk, the database can restore the data based on the redo log to ensure data integrity and consistency.
In other words, the write-before log mechanism ensures data persistence, while the Buffer Pool is used to improve query performance, cache frequently accessed data pages, and reduce I/O operations on disk. The two work together to ensure data security and high-performance operation of the database.
Under normal circumstances, MySQL will flush the data in the Buffer Pool to disk according to a certain strategy to ensure data persistence.
Under normal circumstances, MySQL will periodically flush the dirty pages in the Buffer Pool (data pages that have been modified but not yet written to the disk) to the disk to ensure data persistence. The refresh strategy can be adjusted through configuration parameters, such as innodb_max_dirty_pages_pct and innodb_io_capacity, etc.
Redo Log
In a nutshell: Reduce random disk writes and use memory for high-speed sequential writes. Provide crash data recovery mechanism.
Redo Log logs provide mysql with crash-safe capabilities, that is, crash recovery capabilities.
We know that when we use mysql, its performance bottleneck lies in random disk IO operations, so if we can reduce disk IO operations, it can greatly improve performance.
When we use the addition, deletion, and modification operations, it is actually a random IO operation. If we read the corresponding operation data from the disk every time and then modify and write it, then the performance can be imagined to be low.
So, is there a way that we can write the data to the cache (memory) first, and then write the data to the disk at the right time?
Yes, it is what we call Redo Log.
When we need to write data, we don't write the data directly to disk, but first write it to a faster memory file such as redo log, so the impact on performance will be much smaller.
What is recorded in the redo log is the operation of the data block in mysql.
When our mysql crashes, when we restart to restore data, we can write the data that has not been written to the disk from the redo log to him, thus ensuring data consistency.
Bin Log
In a word: provide data recovery, data rollback, master-slave synchronization and other functions.
Binlog (binary log) is a log file in MySQL that is used to record all modification operations of the database, such as insert, update, and delete. Binlog has the following important functions and meanings:
Data recovery and backup: Binlog can be used for data recovery and backup. By applying the Binlog file to the MySQL instance, the database can be restored to a specific point in time or a specific transaction state. This is very useful in case of data loss or database crash.
Master-slave replication: Binlog is the basis of MySQL master-slave replication. In master-slave replication, all modification operations on the master database will be recorded in the Binlog file and transmitted to the slave database through the network. The slave database performs the same modification operation according to the content of the Binlog file, thereby maintaining data consistency with the master database.
Database synchronization and high availability: Binlog can be used to propagate change operations to other MySQL instances in real time, thereby achieving database synchronization and high availability. By transferring the Binlog file to other MySQL instances, these instances can apply change operations to their own databases to maintain data consistency.
Data analysis and query playback: Binlog records all modification operations of the database, so it can be used for data analysis and query playback. By analyzing the Binlog file, you can understand the historical modification operations of the database, and perform performance analysis and data statistics. For scenarios where historical queries need to be replayed, the Binlog file can be applied to the test environment to simulate query execution.
In MySQL, Bin Log is a binary log used to record database change operations. The following are the operations that will be written to the Bin Log log:
DML statement (Data Manipulation Language): Including INSERT, UPDATE and DELETE statements, used to add, delete, and modify data in the table.
DDL statement (Data Definition Language): including CREATE, ALTER and DROP statements, used to create, modify and delete database objects, such as tables, indexes, etc.
DCL statement (Data Control Language): including GRANT and REVOKE statements, used to authorize and revoke permissions.
Data modification functions and stored procedures: If the functions or stored procedures contain operations that modify data, the execution of these functions or stored procedures will also be written to the Bin Log log.
The purpose of writing to Bin Log is to achieve database persistence and data recovery. By recording change operations into Bin Log, data recovery and synchronization can be performed in case of failure or data loss. In addition, Bin Log logs can also be used in scenarios such as database replication, data backup, and data migration.
Undo Log
In a word: the data that can be seen by the view of MVCC multi-version concurrency control depends on this.
Undo log (rollback log) is a part of MySQL InnoDB storage engine, which is used to realize the atomicity and consistency of transactions. Its function is to record the reverse operation of the transaction operation, so that when the transaction is rolled back or the system crashes, it can be restored to the state before the transaction started.
When a transaction begins, InnoDB creates an undo log for the transaction. During transaction execution, if data modification (such as insert, update, delete) occurs, the corresponding reverse operation will be recorded in the undo log, so that these modifications can be undone when rolling back.
Specifically, for insert operations, the undo log records the inserted data and where the data needs to be deleted; for update and delete operations, the undo log records the modified or deleted data and how to restore it to its original state.
When a transaction needs to be rolled back, InnoDB will undo the previous data modification according to the reverse operation in the corresponding undo log, and restore it to the state before the transaction started. This ability is one of the basic characteristics of transactions, which guarantees the atomicity and consistency of transactions.
For the performance impact of MySQL, the undo log will occupy a certain amount of storage space. The undo log of each transaction needs to be written to disk first, which may cause certain IO overhead. At the same time, when transactions are executed concurrently, the management and reading and writing of the undo log will also cause some additional overhead. Therefore, in high-concurrency scenarios, it is necessary to set the size and management strategy of the undo log reasonably to ensure the balance of system performance.
Undo log is closely related to multi-version concurrency control (MVCC) in the MySQL InnoDB storage engine, and plays an important role in MVCC. MVCC is a concurrency control mechanism used to achieve transaction isolation in high concurrency environments.
MVCC manages concurrent access of transactions by assigning a unique transaction ID (Transaction ID) and version number to each transaction. When a transaction starts, the snapshot version in the current database is recorded. As the transaction progresses, other transactions can continue to read the database and read the snapshot version of the data without being affected by the modifications made by the current transaction.
This introduces the role of undo log. In MVCC, when a transaction is modifying data, in order to ensure that other transactions can read consistent data, InnoDB will copy the data before modification to the undo log, and record the operation in the undo log version number.
When other transactions read data, if the data is modified by the transaction being modified, InnoDB will restore the data to the state before the modification through the rollback operation according to the version number of the transaction and the information in the undo log, and then read Taking the restored data ensures the consistency of reading.
Therefore, the role of undo log in MVCC is to provide a historical version for restoring data to ensure that the data read by a transaction is consistent. It is used to implement data rollback and rollback segment management, providing the necessary support for MVCC.
It should be noted that MVCC not only relies on undo log, but also closely combines with other mechanisms (such as read view, rollback segment, etc.) to achieve transaction isolation and concurrency control. As part of it, the undo log plays a key role.
The collaboration process of these logs
According to whether the data corresponding to the modification request is in memory, the execution process of operating these logs is as follows:
Change Buffer: First, the modification operation will be recorded in the Change Buffer of the memory. Change Buffer is a mechanism of MySQL to delay the actual modification of the corresponding page on disk to improve performance. Modification operations are recorded in the Change Buffer, rather than immediately written to the corresponding page on disk.
Buffer Pool: The main storage location of data in memory is Buffer Pool (also known as page cache). When a modification operation needs to read a data page, MySQL will first check whether there is a page to be modified in the Buffer Pool. If it exists, modify it directly in the Buffer Pool. If the page to be modified is not in the Buffer Pool, the next time we query this data, we will load it into the Buffer Pool and use the modification operation in the Change Buffer to operate on it.
Undo Log: During the execution of the modification operation, MySQL will record the modification operation on the original data to the Undo Log. Undo Log is used to roll back transactions or restore data to previous versions. By recording the data before modification, MySQL can undo or roll back the modification operation when needed.
Redo Log: At the same time, the log of the modification operation will also be written to the Redo Log. Redo Log is an important log for crash recovery. It records all modification operations to ensure data persistence even under abnormal conditions.
Bin Log: Finally, modification operations are also recorded in Bin Log (binary log). Bin Log is a log for replication and recovery. By recording modification operations, data can be synchronized between master and slave servers, and can be used to restore data in disaster recovery.
Note that the final process of writing Bin Log and Redo Log involves two-phase commit.
That is to say, no matter whether the Change Buffer is used or not, as long as we modify the operation, we will write the data to the redo log