MySQL transaction log

1 Introduction

Transactions have 4 characteristics:Atomicity, consistency, isolation and durability. So on what mechanism are the four characteristics of transactions based on?
practical==isolationreasonlock system == Reality.
And ==The atomicity, consistency and durability of the transaction are guaranteed by the redo log and undo log of the transaction==.
- REDO LOG is calledredo log, which provides rewrite operations and restores page operations modified by committed transactions,Used to ensure transaction durability。
- UNDO LOG is calledrollback log, rollbackRows logged to a specific version,Used to ensure transactionatomicity and consistency。
Some DBAs may think that UNDO is the reverse process of REDO, but it is not. Both REDO and UNDO can be regarded as a recovery operation, but
- redo log: is the log generated by the storage engine layer (innodb), which records " ;Physical level"Page modification operation
  - For example, the page number xxx and offset yyy are written with 'zzz' data. Mainly to ensure the reliability of data;
- undo log: is the log generated by the storage engine layer (innodb), which records the logic of Operation log
  - For example, if an INSERT statement is performed on a certain row of data, the undo log will record an opposite DELETE operation.
  - is mainly used for rollback of transactions (undo log records the reverse operation of each modification operation ) and < a i=3>Consistent non-locking read (undo log rollback row record to a specific version - MVCC, that is, multi-version concurrency control).

2. redo Nishi

The InnoDB storage engine manages storage spacein page units. Before actually accessing the page, the page on the disk needs to be cached into the Buffer Pool in the memory. It can only be accessed after >. All changes mustfirst update the data in the buffer pool, and then dirty pages), and the gap between the CPU and the disk is optimized through the buffer pool , so as to ensure that the overall performance does not drop too fast. checkPoint mechanism will be flushed to the disk at a certain frequency (

Why REDO logs are needed

Indatabase management system,REDO log is used to ensure the durability of data. A mechanism. When the database performs write operations, the REDO log records the detailed information of these operations to ensure that even if abnormal situations such as system crashes occur, the database can correctly restore these operations during recovery.

Specifically, when the database performs write operations, itfirst writes these operations to the REDO log and then writes them to disk. This means that even if a system crash occurs before writing to disk, the database can use the information in the REDO log to resume the write operation.

In addition, REDO logs can also be used to achieve transaction durability and consistency. Before the transaction is committed, all modification operations will be recorded in the REDO log and then written to disk. If an abnormality such as a system crash occurs before a transaction is committed, the database can use the REDO log to recover uncommitted transactions to ensure data consistency and integrity.

Therefore, REDO logs are very important for the security and reliability of the database and are an indispensable part of the database system.

Benefits and features of REDO logs

benefit

The redo log reduces the frequency of disk flushing
The redo log takes up very little space

To store the table space ID, page number, offset and the value that needs to be updated, the storage space required is very small and disk flushing is fast.

Features

Redo logs are written to disk sequentially
- In the process of executing a transaction, each time a statement is executed, several redo logs may be generated.These logs are written to the disk in the order in which they are generated< a i=2>, that is, using sequential IO, which is faster than random IO.
During transaction execution, redo log continuously records
- The difference between redo log and bin log,redo log is generated by the storage engine layer, while bin log is generated by the database layer.
- Suppose a transaction inserts 100,000 rows of records into the table. During this process, records are continuously recorded sequentially in the redo log, but the bin log will not record them. It will not be written to the bin log file until the transaction is committed. .

The composition of redo

Redo log can be simply divided into the following two parts:

Redo log buffer (redo log buffer), stored inmemory, is easy lost.

Parameter settings: innodb_log_buffer_size:

Redo log buffer size, the default is 16M, the maximum value is 4096M, and the minimum value is 1M.

mysql> show variables like '%innodb_log_buffer_size%';
+------------------------+----------+
| Variable_name | Value |
+------------------------+----------+
| innodb_log_buffer_size | 16777216 |
+------------------------+----------+

       
       
        
        

       
       
        
        
         
         1
         
         2
         
         3
         
         4
         
         5
         
         6

redo log file (redo log file), stored inhard disk, is persistent .

The REDO log file is as shown in the figure, among which ib_logfile0 and ib_logfile1< a i=4> is the REDO log.

The overall process of redo

Taking an update transaction as an example, the redo log flow process is as shown in the following figure:

Step 1: First read the original data from the disk into the memory and modify the memory copy of the data.
Step 2: Generate a redo log and write it into the redo log buffer. What is recorded is the modified value of the data.
Step 3: When the transaction commits, refresh the contents in the redo log buffer to the redo log file, and use append writing to the redo log file.
Step 4: Periodically refresh the modified data in memory to disk

Redo log brushing strategy

The redo log is not written directly to the disk.The InnoDB engine will first write the redo log buffer when writing the redo log , and then flush it into the real redo log file at a certain frequency.

Note,The process of flushing the redo log buffer to the redo log file is not the actual flushing to the disk,Just flush it into the file system cache (page cache) (This is an optimization made by modern operating systems to improve file writing efficiency), the actual writing will be handed over to The system decides by itself (for example, the page cache is large enough) . Then there is a problem for InnoDB. If it is left to the system for synchronization, if the system goes down, the data will also be lost (although the probability of the entire system going down is still relatively small).

In this case, InnoDB givesinnodb_flush_log_at_trx_commit parameter, which controls commit in. It supports three strategies:redo log file toredo log bufferWhen committing a transaction, how to refresh the log in

0: means **When the transaction is committed, the Redo Log is not flushed to the disk. Instead, the unflushed Redo Log is flushed to the disk in batches every second.**. This is the highest performance refresh strategy, but if the MySQL process crashes or the machine loses power, a second of data may be lost.
1（默认）:express**Redo Log is written to disk synchronously every time a transaction is committed.. This isthe safest refresh strategy**, but the performance is poor because it needs to wait for each transaction to be committed before the Redo Log is written to disk.
2: means **Write the Redo Log to the system cache every time a transaction is committed (not disk), but the system will flush the Redo Log in the cache to disk in batches every second**. This is a balance between performance and security, which can ensure that the Redo Log is flushed to disk at least every second while avoiding the submission of each transaction. There is a performance loss when writing to disk.

Note: The background thread will write the contents of the redo log buffer to the page cache every second, and then call fsync to flush the disk.

3. Undo Japan

Redo log is a guarantee of transaction durability, and undo log is a guarantee of transaction atomicity.. In the transactionupdating data the precursor operation actually needs to be written first Enter aundo log.

How to understand Undo logs

Transactions need to guarantee atomicity , that is, all operations in the transaction are either completed or nothing is done. But sometimes some situations may occur in the middle of transaction execution, such as:

Situation 1: Various errors may be encountered during transaction execution, such aserrors on the server itself, operating system errors, or even sudden Power failure caused the error.
Case 2: The programmer can manually enter the ROLLBACK statement during transaction execution to end the execution of the current transaction.

When the above situation occurs, we need to change the data back to its original appearance. This process is calledrollback, this can create an illusion: this transaction seems to have done nothing, so it meets the atomicity requirement.

Whenever we want to make changes to a record (the changes here can refer to INSERT, DELETE, UPDATE), all need to "keep a hand" - write down what is needed when rolling back. For example:

When you insert a record, at least write down the primary key value of this record. When you roll back later, you only need to Just delete the record corresponding to this primary key value. (For each INSERT, the InnoDB storage engine will complete a DELETE)
You have deleted a record. At least write down the contents of this record, so that when you roll back later, you can delete the record. Just insert the record composed of these contents into the table. (For each DELETE, the InnoDB storage engine will execute an INSERT)
Youmodified a record. At least record all the old values before modifying this record, so that you can roll back later. Just update this record to the old value. (For each UPDATE, the InnoDB storage engine will perform a reverse UPDATE and put back the row before modification)

MysQL calls these contents recorded for rollback **Undo log or rollback log (i.e. undo log)**。

Note that since the query operation ( SELECT ) does not modify any user records, so in When the query operation is executed, the corresponding undo log does not need to be recorded.

In addition, undo log will generate redo log, that is, the generation of undo log will be accompanied by The redo log is generated becauseUndo log also requires persistence protection。

The role of undo logs

Function 1:Rollback data
- undo is a logical log, sojust logs the databaseLogically return to the original state. All modifications are logically undone, but the data structures and pages themselves may look very different after the rollback.The database is not physically restored to the way it was before the statement or transaction was executed.
Function 2:MVCC
- Another function of undo is MVCC, that is, the implementation of MVCC in the InnoDB storage engine is completed through undo. When a user reads a row of records, if the record is already occupied by other transactions, the current transaction can read the previous row version information through undo to achieve non-locking reading.

The storage structure of undo

Rollback segment and undo page

InnoDB uses segments to manage undo logs, that is,rollback segment (rollback segment). Each rollback segment records1024 undo log segments, and is performed in each undo log segment Application for undo page. The Undo page is a memory cache area used to store modified data during transaction processing.

BeforeInnoDB1.1 version (excluding version 1.1), there was only one rollback segment, so the limit of simultaneous online transactions was < /span>. Although it is sufficient for most applications. 1024
Starting from version 1.1, InnoDB supports a maximum of 128 rollback segments, so its simultaneous online transaction limit is increased to < a i=3>128*1024.

mysql> show variables like 'innodb_undo_logs '; # 可以看出 value 为 128

   
   
    
    

   
   
    
    
     
     1

Although InnoDB version 1.1 supports 128 rollback segments, these rollback segments are stored in the shared table space ibdata. Starting from InnoDB version 1.2, the rollback segment can be further set through parameters. These parameters include:

innodb_undo_directory: Set the path where the rollback segment file is located. This means that the rollback segment can be stored in a location other than the shared table space, that is, it can be set as an independent table space. The default value of this parameter is ".times", which represents the directory of the current InnoDB storage engine.
Innodb_undo_logs: Set the number of rollback segments. The default value is 128. In InnpDB1.2 version, this parameter is used to replace the parameter innodb_rollback_segments of the previous version.
innodb_undo_tablespaces: Set the number of files that make up the rollback segment so that the rollback segment can be evenly distributed among multiple files. After setting this parameter, you will see a file prefixed with undo in the path innodb_undo_directory, which represents the rollback segment file.

undo log related parameters are generally rarely changed.

Reuse of undo pages

When we start a transaction and need to write undo log, we must first find a free position in the undo log segment. When there is a free space, we apply for an undo page and perform undo log in the applied undo page. of writing. We know that the default page size of mysql is 16k.

Allocating a page for each transaction is very wasteful (unless your transaction is very long). Assume that the TPS (number of transactions processed per second) of your application is 1000, then 1000 pages are needed in 1s, which is about 16M. Storage, approximately 1G of storage is required per minute. If this continues, unless MySQL cleans up very diligently, the disk space will grow very quickly over time, and a lot of space will be wasted.

So the undo page is designed to be reusable. When the transaction is committed, the undo page will not be deleted immediately. Because of reuse, this undo page may be mixed with undo logs of other transactions. After the undo log is committed, it will be placed in a linked list, and then it will be judged whether the space used by the undo page is less than 3/4. If it is less than 3/4, it means that the current undo page can be reused, then it will not be recycled. The undo log of other transactions can be recorded in the current undo page. later. Since the undo log isdiscrete, it is not efficient to clean up the corresponding disk space.

Rollback segments and transactions

Each transaction will only use one rollback segment, and one rollback segment may serve multiple transactions at the same time.
When a transaction starts, a rollback segment will be created. During the transaction, when the data is modified, the original data will be copied to the rollback segment.
In a rollback segment, transactions continue to fill extents until the transaction ends or all space is used up. If the current extent is not enough, the transaction will request the expansion of the next extent in the segment. If all allocated extents are used, the transaction will overwrite the original extent or expand the new extent if the rollback segment allows. panels to use.
Rollback segments exist in undo table spaces. Multiple undo table spaces can exist in the database, but only one undo table space can be used at the same time.

mysql> show variables like 'innodb_undo_tablespaces ' ;

#The number of undo logs is at least 2. The truncate operation of undo log is initiated by the purge coordination thread. During the process of truncate an undo log table space, it is guaranteed that there is an available undo log available.

When a transaction commits, the InnoDB storage engine does the following two things:
- Put the undo log into the list for subsequent purge operations
  - The purge operation is actually to support MySQL's MVCC, so the record cannot be processed immediately when the transaction is submitted, because other transactions may be using this row, so the InnoDB storage engine needs to save the previous version of the record.
  - If the row record is no longer referenced by any other transaction, then a real delete operation can be performed. Therefore, it can be understood that the purge operation is not a delete operation to process the current transaction, but to clear the previous delete and update operations. In fact, The only operation performed is delete, which cleans the version recorded in the previous row.
- Determine whether the page where the undo log is located can be reused, and if it can be allocated to the next transaction,

Data classification in rollback segments

Uncommitted rollback data (uncommitted undo information): The transaction associated with this data has not been committed and is used to achieve read consistency. So this data cannot be overwritten by data from other transactions.
Committed but not expired rollback data (committed undo information): The transaction associated with this data has been committed, but it is still affected by the retention time of the undo retention parameter.
Data that the transaction has submitted and expired (expired undo information): The transaction has been submitted, and the transaction storage time has exceeded the time specified by the undo retention parameter, which belongs to expired data. When the rollback segment is full, "data that has been committed and expired by the transaction" will be overwritten first.

The undo log and the page where the undo log is located cannot be deleted immediately after the transaction is committed. This is because there may be other transactions that need to go through undo log to get the previous version of the row record. When the story transaction is submitted, the undo log is put into a linked list. Whether the undo log can be finally deleted and the page where the undo log is located are judged by the purge thread.

Type of undo

insert undo log
- insert undo log refers to the undo log generated during the insert operation. Because the record of the insert operation is only visible to the transaction itself and not to other transactions (this is a requirement for transaction isolation), the undo log< /span>. can be deleted directly after the transaction is committed. No purge operation is required
update undo log
- update undo log records the undo log generated by delete and update operations. The undo log may need to provide an MVCC mechanism , so it cannot be deleted when the transaction is committed< a i=4>. When submitting, put it into the undo log list, and wait for the purge thread to perform the final deletion.

The life cycle of undo log

Brief generation process

The following is the simplified process of undo+redo transaction
Assume there are 2 values, A=1 and B=2, and then modify A to 3 and B to 4

1. start transaction ;
2．记录 A=1到undo log;
3. update A = 3;
4．记录A=3 到redo log;
5．记录B=2到undo log;
6. update B = 4;
7．记录B =4到redo log;
8．将redo log刷新到磁盘;
9. commit

 
 
  
  

 
 
  
  
   
   1
   
   2
   
   3
   
   4
   
   5
   
   6
   
   7
   
   8
   
   9

If the system crashes in any step 1-8 and the transaction is not committed, the transaction will not have any impact on the data on the disk.
If it goes down between 8 and 9, you can choose to roll back after recovery, or you can choose to continue to complete the transaction submission, because the redo log has been persisted at this time.
If the system crashes after 9, and the changed data in the memory map has not had time to be flushed back to the disk, then after the system is restored, the data can be flushed back to the disk according to the redo log.

Only Buffer Pool process:

[External link image transfer failed. The source site may have an anti-leeching mechanism. It is recommended to save the image and upload it directly (img-tyHXqwlM-1680921939349) (C:/Users/dell/AppData/Roaming/Typora/typora-user-images/ image-20230327193914352.png)]

redo + undo

Before updating the data in the Buffer Pool, we need to write the status before the start of the data transaction to the Undo Log. If something goes wrong halfway through the update, we can use Undo Log to roll back to before the transaction started.

Detailed generation process

For the InnoDB engine, in addition to the data of the record itself, each row record also has several hidden columns:

DB_ROW_ID: If the primary key is not explicitly defined for the table, and there is no unique index defined in the table, InnoDB will automatically add a hidden column of row_id to the table as the primary key.
DB_TRX_ID: Each transaction will be assigned a transaction ID. When a record is changed, the transaction ID of this transaction will be written into trx_id.
DB_ROLL_PTR: The rollback pointer is essentially a pointer to the undo log.

When executing INSERT

begin;
INSERT INTO user (name) VALUES ("tom");

 
 
  
  

 
 
  
  
   
   1
   
   2

The inserted data will generate an insert undo log, and the rollback pointer of the data will point to it. The undo log will record the serial number of the undo log, the column and value of the inserted primary key..., then when performing rollback, the corresponding data can be deleted directly through the primary key.

When executing UPDATE

For update operations, update undo log will be generated, and the primary key will be updated and the primary key will not be updated. Suppose you execute now:

UPDATE user SET name="Sun" WHERE id=1;

 
 
  
  

 
 
  
  
   
   1

At this time, the old record will be written to the new undo log, so that the rollback pointer points to the new undo log, its undo no is 1, and the new undo log will point to the old undo log (undo no=0).

Suppose you execute now:

UPDATE user SET id=2 WHERE id=1 ;

 
 
  
  

 
 
  
  
   
   1

**For the operation of updating the primary key, the original data deletemark will be opened first. At this time, the data is not actually deleted. **The real deletion will be left to the cleaning thread to judge,< a i=1> Then insert a new piece of data later. The new data will also generate an undo log, and the sequence number of the undo log will increase.

It can be found that every change to data will generate an undo log. When a record is changed multiple times, multiple undo logs will be generated. The undo log records the log before the change, and the sequence number of each undo log It is incremental, so when you want to roll back, push forward according to the sequence number, and you can find our original data.

How to roll back undo log

Taking the above example, assuming rollback is executed, the corresponding process should be as follows:

Delete the data with id=2 through undo no=3 log
Restore the deletemark of the data with id=1 to 0 through the log of undo no=2
Restore the name of the data with id=1 to Tom through the log of undo no=1
Delete the data with id=1 through undo no=0 log

Deletion of undo log

For insert undo log

Because the record of the insert operation is only visible to the transaction itself and not to other transactions. Therefore, the undo log can be deleted directly after the transaction is submitted, and no purge operation is required.

For update undo log

The undo log may need to provide an MVCC mechanism, so it cannot be deleted when the transaction is committed. When submitting, put it into the undo log list and wait for the purge thread to perform the final deletion.

The two main functions of the purge thread are:

Clean up the undo page and clear the data rows with the Delete_Bit identifier in the page. In InnoDB, the Delete operation in a transaction does not actually delete the data row, but a Delete Mark operation that marks the Delete_Bit on the record without deleting the record. It is a kind of "fake deletion"; it is just marked. The real deletion work requires the background purge thread to complete.

3. Summary

redo log is a physical log, records the physical changes of the data page, undo log is not the reverse process of redo log.
undo log is a logical log,When rolling back a transaction, the database is only logically restored to its original state.

reference

https://www.bilibili.com/video/BV1iq4y1u7vj/?spm_id_from=333.337.search-card.all.click&vd_source=25b05e9bd8b4bdac16ca2f47bbeb7990