Overview
The main purpose of the database system to introduce transactions: the transaction will transform the database from a consistent state to another consistent state. When submitting the work to the database, you can ensure that either all the changes have been saved, or all the changes are not saved.
Know the affairs
A transaction can be composed of a very simple SQL statement or a group of complex SQL statements. A transaction is a program execution unit that accesses and updates various data items in the database. The operations in the transaction are either modified or not done. This is the purpose of the transaction and one of the important characteristics of the transaction model that distinguishes it from the file system.
For the InnoDB storage engine, the default transaction isolation level is READREPEATABLE,
and the transactions in the InnoDB storage engine fully comply with the characteristics of ACID. ACID is an abbreviation of the following 4 words:
- Atomicity
Atomicity means that the entire database transaction is an indivisible unit of work. Only when all database operations in the transaction are executed successfully can the entire transaction be considered as successful. If any SQL statement in the transaction fails to execute, the SQL statement that has been successfully executed must also be cancelled, and the database state should return to the state before the transaction was executed.
- Consistency
Consistency means that a transaction transforms the database from one state to the next consistent state. Before the start of the transaction and after the end of the transaction, the integrity constraints of the database have not been destroyed.
- Isolation
There are other names for isolation, such as concurrency control, serializability, and locking. The isolation of the transaction requires that the object of each read-write transaction can be separated from the operating objects of other transactions, that is, the transaction is invisible to other transactions before the transaction is committed. This is usually achieved by using locks. Current database systems provide a granular lock strategy that allows transactions to lock only a subset of entity objects, thereby improving the concurrency between transactions.
- Durability
Once the transaction is committed, the result is permanent. Even if a failure such as a downtime occurs, the database can recover data. It should be noted that the permanence of the result can only be guaranteed from the perspective of the transaction itself. For example, after the transaction is committed, all changes are permanent. Even when the database needs to be restored due to a crash, it can be guaranteed that the submitted data will not be lost after the restoration. However, if it is not the failure of the database itself, but some external reasons, such as damage to the RAID card, natural disasters, etc., that cause the database to have problems, then all the submitted data may be lost.
Transaction classification
From the perspective of transaction theory, transactions can be divided into the following types:
- Flat Transactions (Flat Transactions)
Flat transaction (Flat Transaction) is the simplest type of transaction, but in the actual production environment, this may be the most frequently used transaction. In a flat transaction, all operations are at the same level, which start from BEGINWORK and end by COMMIT WORK or ROLLBACK WORK. The operations in between are atomic, either all are executed or all are rolled back. Therefore, flat transactions are the basic building blocks for applications to become atomic operations.
The main limitation of flat transactions is that they cannot commit or roll back a certain part of the transaction, or commit in several steps.
If a planned rollback operation is supported, then the entire transaction does not need to be terminated. So there is a flat transaction with save points.
- Flat transactions with savepoints (Flat Transactions with Savepoints)
In addition to supporting operations supported by flat transactions, it is allowed to roll back to an earlier state in the same transaction during transaction execution. This is because errors that may occur during the execution of certain transactions will not cause all operations to be invalid, and it is not satisfactory to give up the entire transaction, and the overhead is too large. Savepoint is used to inform the system that the current state of the transaction should be remembered so that when an error occurs later, the transaction can return to the state at the savepoint.
- Chained Transactions
Chained Transaction can be regarded as a variant of the savepoint model. For flat transactions with savepoints, when the system crashes, all savepoints will disappear, because the savepoints are volatile, not persistent (persistent). This means that when recovering, the transaction needs to be re-executed from the beginning, and cannot be continued from the most recent save point.
The idea of chain transaction is; when a transaction is submitted, the data object that is not needed is released, and the necessary processing context is implicitly passed to the next transaction to be started. Note that the commit transaction operation and the start of the next transaction operation will be merged into one atomic operation. This means that the next transaction will see the results of the previous transaction, as if it were in a transaction.
Chain transactions are different from flat transactions with savepoints in that flat transactions with savepoints can be rolled back to any correct savepoint. The rollback in the chain transaction is limited to the current transaction, that is, it can only be restored to the most recent save point. For the lock processing, the two are different. The chain transaction releases the lock held by the current transaction after the COMMIT is executed, and the flat transaction with savepoint does not affect the lock held so far.
- Nested Transactions
Nested Transaction (Nested Transaction) is a hierarchical framework. A top-level transaction <top-level transaction) controls all levels of transactions. The transactions nested under the top-level transaction are called subtransactions, which control each local transformation.
The following gives Moss's definition of nested transactions:
1) A nested transaction is a tree composed of several transactions. The subtree can be either a nested transaction or a flat transaction.
2) The transaction at the leaf node is a flat transaction. But the distance from the root to the leaf node of each sub-transaction can be different.
3) The transaction at the root node is called the top-level transaction, and other transactions are called sub-transactions. The predecessor of the transaction is called the parent, and the next level of the transaction is called the child.
4) Sub-transactions can either be committed or rolled back. But its commit operation does not take effect immediately, unless its parent transaction has been committed. Therefore, it can be inferred that any sub-things are actually submitted after the top-level transaction is submitted.
5) The rollback of any transaction in the tree will cause all its sub-transactions to roll back together, so the sub-transactions only retain the characteristics of A, C, and I, and do not have the characteristics of D.
In Moss's theory, the actual work is done by the leaf nodes, that is, only the transactions of the leaf nodes can access the database, send messages, and obtain other types of resources. The high-level transaction is only responsible for logic control and decides when to call related sub-transactions. Even if a system does not support nested transactions, users can also simulate nested transactions through savepoint technology.
- Distributed Transactions ((Distributed Transactions)
Distributed Transactions are usually flat transactions that run in a distributed environment, so different nodes in the network need to be accessed according to the location of the data.
Suppose a user performs a bank transfer operation on an ATM, for example, a cardholder transfers 10,000 yuan from a savings card of China Merchants Bank to a savings card of ICBC. In this case, the ATM machine can be regarded as node A, the back-end database of China Merchants Bank can be regarded as node B, and the back-end database of ICBC can be regarded as C. This transfer operation can be decomposed into the following steps:
1) Node A sends out Transfer order.
2) Node B subtracts 10 000 from the balance value in the savings card.
3) Node C adds 10 000 to the balance value in the savings card.
4) Node A notifies the user that the operation is complete or node A notifies the user that the operation has failed.
Distributed transactions need to be used here, because node A cannot complete the task by calling a database. It needs to access the databases of two nodes in the network, and the transaction operations performed in the database of each node are flat. For distributed transactions, it also needs to meet the ACID characteristics, either all happen or all fail. For the above example, if any of the operations in steps 2) and 3> fails, the entire distributed transaction will be rolled back. If not, the result would be terrible.
For the InnoDB storage engine, it supports flat transactions, transactions with save points, chain transactions, and distributed transactions. For nested transactions, it does not natively support, therefore, MySQL database or InnoDB storage engine seems powerless for users with parallel transaction needs. However, users can still simulate serial nested transactions through transactions with savepoints.
Realization of the transaction
Transaction isolation is achieved by locks. The redo log is called the redo log and is used to ensure the atomicity and durability of the transaction. The undo log is used to ensure the consistency of the transaction.
Some DBAs may think that undo is the reverse process of redo, but it is not. The role of redo and undo can be regarded as a kind of recovery operation, redo resumes the page operation that commits the transaction modification, and undo rollback records to a specific version. Therefore, the content recorded by the two is different. Redo is usually a physical log, which records the physical modification operations of the page. Undo is a logical log, which is recorded according to each line of records.
redo
It consists of two parts: one is the redo log buffer in memory, which is volatile; the other is the redo log file, which is persistent.
When the transaction is committed (COMMIT), all logs of the transaction must be written to the redo log file for persistence, and the COMMIT operation of the transaction is completed. The log here refers to the redo log. In the InnoDB storage engine, it consists of two parts, redo log and undo log. The redo log is used to ensure the durability of the transaction, and the undo log is used to help the transaction rollback and MVCC functions. The redo log is basically written sequentially, and there is no need to read the redo log file when the database is running. The undo log requires random read and write.
In the InnoDB storage engine, redo logs are stored in 512 bytes. This means that the redo log cache and redo log files are stored in blocks, which are called redo log blocks, and each block is 512 bytes in size.
The difference between binary log and redo log in MySQL
- First of all, the redo log is generated at the InnoDB storage engine layer, and the binary log is generated at the upper layer of the MySQL database, and the binary log is not only for the InnoDB storage engine, any storage engine in the MySQL database will generate a binary for changes to the database. Log.
- Secondly, the two types of log records have different forms of content. The binary log of the upper layer of the MySQL database is a kind of logical log, which records the corresponding SQL statements. The redo log at the InnoDB storage engine level is a physical format log, which records the modification of each page.
- In addition, the two log records are written to disk at different time points, as shown in Figure 7-6. The binary log is only written once after the transaction is committed. The redo log of the InnoDB storage engine is continuously written during the transaction, which shows that the log is not written in the order in which the transaction is committed.
================================================== =======================
undo
The redo log records the behavior of the transaction, which can be used to "redo" the page. But the transaction sometimes needs to be rolled back, and then undo is needed. Therefore, when the database is modified, the InnoDB storage engine will not only generate redo, but also a certain amount of undo. In this way, if the transaction or statement executed by the user fails for some reason, or the user requests a rollback with a ROLLBACK statement, the undo information can be used to roll back the data to the way it was before the modification.
Redo is stored in the redo log file. Unlike redo, undo is stored in a special segment (segment) inside the database. This segment is called undo segment (undo segment). The undo segment is located in the shared table space.
Users usually have such a misunderstanding about undo: undo is used to physically restore the database to the way it was before executing the statement or transaction-but this is not the case. Undo is a logical log, so it just logically restores the database to its original state. All modifications are logically cancelled, but the data structure and the page itself may be quite different after the rollback. This is because in a multi-user concurrent system, there may be tens, hundreds or even thousands of concurrent transactions. The main task of the database is to coordinate concurrent access to data records. For example, a transaction is modifying a few records in the current page, while other transactions are modifying other records in the same page. Therefore, you cannot roll back a page to the beginning of the transaction, because this will affect the ongoing work of other transactions.
For example, the user performs an INSERT 10W record transaction, this transaction will result in the allocation of a new segment, that is, the table space will increase. When the user executes ROLLBACK, the inserted transaction will be rolled back, but the size of the table space will not shrink as a result. Therefore, when the InnoDB storage engine rolls back, it actually does the opposite of the previous job. For each INSERT, the InnoDB storage engine will complete a DELETE; for each DELETE, the InnoDB storage engine will perform an INSERT; for each UPDATE, the InnoDB storage engine will perform a reverse UPDATE, putting the rows before the modification back.
In addition to the rollback operation, another role of undo is MVCC, that is, the implementation of MVCC in the InnoDB storage engine is done through undo. When a user reads a row of records, if the record is already occupied by other transactions, the current transaction can read the previous row version information through undo to achieve non-locking reading.
The last and most important point is that undo log will generate redo log, that is, the generation of undo log will be accompanied by the generation of redo log, because undo log also needs persistent protection.
purge thread
Purge is used to finally complete delete and update operations. This design is because the InnoDB storage engine supports MVCC, so records cannot be processed immediately when the transaction is committed. At this time, other things may be referencing this line, so the InnoDB storage engine needs to save the previous version of the record. Whether the record can be deleted is judged by purge. If the row record has not been referenced by any other transaction, then the real delete operation can be performed.
group commit
If the transaction is a non-read-only transaction, an fsync (system call provided by the system for flushing the kernel buffer to the disk) operation is required every time the transaction is committed to ensure that the redo log has been written to the disk. When the database is down, it can be recovered through the redo log. Although the appearance of solid-state drives has improved the performance of disks, the fsync performance of disks is limited. In order to improve the efficiency of disk fsync, the current database provides the function of group commit, that is, one fsync can refresh to ensure that multiple transaction logs are written to the file. For the InnoDB storage engine, two phases of operations are performed when a transaction is committed:
1) Modify the information corresponding to the transaction in the memory, and write the log to the redo log buffer.
2) Calling fsync will ensure that all logs are written to disk from the redo log buffer.
About transaction control statements
- Transactions in the InnoDB storage engine are atomic, which shows the following two situations: each statement that constitutes a transaction will be committed (become permanent), or all statements will be rolled back. This protection also extends to individual sentences. A statement is either completely successful or completely rolled back (note that I am talking about statement rollback). Therefore, when a statement fails and throws an exception, it will not cause the previously executed statement to be automatically rolled back. All executions will be retained, and it is up to the user to decide whether to commit or roll back the operation.
- Another easy mistake is ROLLBACK TO SAVEPOINT. Although there is ROLLBACK, it does not really end a transaction. Therefore, even if ROLLBACK TO SAVEPOINT is executed, you need to explicitly run COMMIT or ROLLBACK commands afterwards.
Transaction isolation level
The four isolation levels defined by the SQL standard are:
- READ UNCOMMITTED
- READ COMMITTED
- REPEATABLE READ
- SERIALIZABLE
The transaction isolation level of SERIALIABLE is mainly used for distributed transactions of the InnoDB storage engine.
Distributed transaction
The InnoDB storage engine provides support for XA transactions, and supports the implementation of distributed transactions through XA transactions. Distributed transaction refers to allowing multiple independent transactional resources to participate in a global transaction. Transaction resources are usually relational database systems, but they can also be other types of resources. The global transaction requires that all participating transactions are either committed or rolled back, which improves the original ACID requirements of the transaction. In addition, when using distributed transactions, the transaction isolation level of the InnoDB storage engine must be set to SERIALIZABLE.
XA transactions allow distributed transactions between different databases. For example, one server is a MySQL database, the other is an Oracle database, and there may be another server that is a SQL Server database, as long as each node participates in the global transaction Both support XA transactions. Distributed transactions may be more common in bank system transfers.
XA transaction consists of one or more resource managers (Resource Managers), a transaction manager (Transaction Manager) and an application program (Application Program).
- Resource Manager: Provides methods to access transaction resources. Usually a database is a resource manager.
- Transaction Manager: Coordinating the various transactions participating in the global transaction. Need to communicate with all resource managers participating in the global transaction.
- Application: Define the boundary of the transaction and specify the operations in the global transaction.
In the distributed transaction of the MySQL database, the resource manager is the MySQL database, and the transaction manager is the client connecting to the MySQL server.
Distributed transactions use a two-phase commit method. In the first phase, all nodes participating in the global transaction begin to prepare (PREPARE), telling the transaction manager that they are ready to commit. In the second stage, the transaction manager tells the resource manager to perform ROLLBACK or COMMIT. If any node shows that it cannot be submitted, all nodes are told to roll back. It can be seen that the difference from local transactions is that distributed transactions require one more PREPARE operation. After receiving the consent information of all nodes, the COMMIT or ROLLBACK operation can be performed.
- Internal XA transaction
The distributed transaction discussed earlier is an external transaction, that is, the resource manager is the MySQL database itself. There is another distributed transaction in the MySQL database, which is called internal XA transaction between the storage engine and the plug-in, or between the storage engine and the storage engine.
The most common internal XA transaction exists between binlog and the InnoDB storage engine. Due to the need of replication, most of the current databases have enabled the binlog function. When the transaction is committed, the binary log is written first, and then the redo log of the InnoDB storage engine. The requirements for the above two operations are also atomic, that is, the binary log and the redo log must be written at the same time.
Long business
Long-Lived Transactions, as the name suggests, are transactions that take a long time to execute.
For long transactions, sometimes it can be handled by converting into mini batches. When
an error occurs in a transaction, only part of the data needs to be rolled back, and then the last completed transaction continues, which can greatly reduce unnecessary costs.