MySQL transaction isolation and isolation level principle

Preface

No matter which object-oriented language we learn, in a multi-threaded concurrency environment, multiple threads operate on the same shared resource, which causes data errors in the resource to be called thread safety. Under normal circumstances, locking can handle thread safety issues well.

I wonder if you have thought about it, MySQL is also a software that supports multi-threaded access, but we don’t seem to pay too much attention to thread safety issues in our daily development? In fact, it is not to say that MySQL will not have thread safety problems, but that it is so good that many places have helped us solve it.

Transaction isolation and isolation level

Transaction isolation means that in a concurrent environment, concurrent transactions are isolated from each other, and the execution of one transaction cannot be interfered by other transactions. In other words, no time with concurrent operation of the same transaction data, data integrity have their own space for each transaction , both data manipulation and the use of an internal affairs of other concurrent transactions are isolated , individual transactions between concurrent execution Can't interfere with each other. When multiple transactions are executed at the same time on the database, dirty read, non-repeatable read, and phantom read may occur. In order to solve these problems, the "isolation level" is introduced. the concept of.

SQL standard transaction isolation levels include: read uncommitted (read uncommitted), read committed (read committed), repeatable read (repeatable read) and serialization (serializable). It should be clear that the higher the isolation level, the better the data consistency, but the lower the efficiency . Dirty reads may occur at the lowest isolation level, but if our system does not cause any impact even if dirty reads occur, then we might as well allow dirty reads. The higher the isolation level of the transaction, the more it can ensure the integrity and consistency of the data, but at the same time the greater the impact on concurrent performance. Therefore, we need to find a balance between the two.

Lock mechanism

3 algorithms of row lock

The InnoDB storage engine has three row lock algorithms, which are

• Record Lock: The lock on a single record
• Gap Lock: gap lock, lock a range but not the record itself
• Next-Key Lock: GapLock + Record Lock, lock a range, and lock itself

RecordLock will always lock the index records. If the InnoDB storage engine table is not set up with any index when it is created, then the InnoDB storage engine will use the implicit primary key to lock it.

Next-Key Lock is a locking algorithm that combines Gap Lock and Record Lock, under the Next-Key Lock algorithm. InnoDB uses this locking algorithm for row queries. For example, if there is an index with four values ​​of 10, 11, 13, 20 and 20, the interval in which the index may be Next-Key Locking is: (-∞, 10], (10, 11), (11, 13] , (13, 20], (20, +∞]

The purpose of adopting Next-Key Lock is to solve phantom reads . If transaction T1 has locked the following range through next-key locking

(10, 11]、(11, 13]

When a new record 12 is inserted, the lock range becomes

(10, 11]、(11, 12]、(12, 13]

The traditional auxiliary index uses Next-Key Locking technology to lock, but when the query index contains a unique index, the InnoDB storage engine will optimize the Next-Key Lock, downgrading to Record Lock only locks itself .

Read lock (S lock, shared lock) select k from t where id=1 lock in share mode; Write lock (X lock, exclusive lock) select k from t where id=1 for update;

The requirement of transaction isolation can be achieved through the lock mechanism, so that transactions can work concurrently . Locking improves concurrency, but it brings potential problems. Fortunately, because of the requirements of transaction isolation, locks will only cause three problems. If these three problems can be prevented from occurring, it will not produce concurrent exceptions .

Dirty read

Dirty read refers to that under different transactions, the current transaction can read the uncommitted data of another transaction . Simply put, it can read dirty data. The so-called dirty data actually refers to the modification of the row records in the buffer pool by the transaction, and the data that has not yet been committed. If dirty data is read in a transaction, it is obviously a violation of database isolation.

Dirty reads do not often occur in the current production environment. The condition for dirty reads to occur is that the transaction isolation level is "read uncommitted".

Non-repeatable

Non-repeatable reading refers to reading the same data set multiple times in a transaction. Before the transaction is over, because another transaction also accesses the same data set and performs some DML operations. As a result, the data read twice in the first transaction may be different due to the modification of the second transaction. In this way, the data read twice in one transaction is different.

Phantom reading problem

A phantom read refers to the fact that when a transaction queries the same range twice before and after, the latter query sees rows that the previous query did not see .

One more explanation is needed for the phantom reading:

1. Under the repeatable read isolation level, ordinary queries are snapshot reads, and you will not see other transactions insert data. Therefore, phantom reading will only appear under "current reading" (for update)
2. The phantom read only refers to the same range of the transaction query, the "newly inserted row" is found in the last query, and the existing data is not considered a phantom read in the second query

Row lock can only lock the row, but the action of "newly inserting a record" is to update the "gap" between records. So in order to solve the phantom read, only one interval is locked. Locking the table is equivalent to locking the entire interval, of course, but the cost is too high. So InnoDB introduced a new lock, which is the gap lock (Gap Lock).

Gap lock, as the name implies, locks the gap between two values. As shown in the table below, when you insert 5 records, there are 6 gaps in the table.

So when you execute select * from t where d=5 for updateit, you not only add row locks to the existing records in the database, but also add gap locks to the gaps in the amount of records, which ensures that no new records can be inserted. In other words, at this time, in the process of scanning row by row, not only the row lock is added to the row, but also the gap lock is added to the gap on both sides of the row.

Deadlock caused by gap lock

Our project usually has such a requirement: arbitrarily lock a row, insert if this row does not exist, update its data if this row exists, the code is as follows:

begin;
select * from t where id=N for update;

/* If the line does not exist*/
insert into t values(N, N, N);
/* If the line exists*/
update t set d=N set id=N;

commit;

There seems to be nothing wrong with this logic. I use for update to lock it up every time I operate. But as long as it encounters concurrency, it will deadlock. The specific scenarios are as follows:

select A	select B
begin; select * from t where id = 9 for update;
	begin; select * from t where id = 9 for update;
	insert into t values(9, 9, 9); (blocked)
insert into t values(9, 9, 9); (ERROR 1213(40001): Deadlock found)

We can see that there is a deadlock without going to the update statement. We analyze it in the order of statement execution:

1. selecn A executes the select ... for update statement. Since the line id=9 does not exist, the gap lock (10, 15) will be added;
2. Select B executes select...for update. Since the row id=9 does not exist, gap locks (10, 15) will also be added. There will be no conflicts between gap locks, so this statement can be executed successfully.
3. Select B tried to illustrate (9,9,9), but was blocked by the gap lock of select A, and had to wait
4. select A tried to insert a row (9,9,9), blocked by the gap lock of select B

Therefore, the two sessions enter a mutual waiting state, forming a deadlock. Of course, InnoDB's deadlock detection immediately discovered this pair of deadlock relationships, causing the insert statement of session A to report an error and return. So if you use "Current Read" to lock the unique index, when the value that needs to be locked does not exist, the gap lock will be used. At this time, the gap lock is a read lock, which means that the gap lock can only protect you from the gap. Insert value .

In the case of repeatable reading, a gap lock will occur. The introduction of gap locks may cause the same statement to lock a larger range, which actually affects the degree of concurrency . Therefore, if you set the isolation level to read commit, there is no gap lock. But at the same time, if you want to solve the possible data and log inconsistency, you need to set the binlog format to row. This is also the configuration combination used by many companies now.

The answer to this question itself is that if the read submission isolation level is sufficient, that is, the business does not need repeatable read guarantees, so considering that the lock range of the operation data under read submission is smaller (no gap lock, only lock rows ) , this choice is reasonable. But at the same time, if you want to solve the possible data and log inconsistency, you need to set the binlog format to row .

Locking rules

The gap lock is only valid under the repeatable read isolation level, without special instructions, the default repeatable read

1. Principle 1: The basic unit of locking is next-key lock. I hope you remember that next-key lock is the interval between front opening and closing.
2. Principle 2: Objects accessed during the search process will be locked.
3. Optimization 1: Equivalent query on the index, when the unique index is locked, next-key lock degenerates into row lock .
4. Optimization 2: Equivalence query on the index. When traversing to the right and the last value does not meet the equivalence condition, the next-key lock degenerates into a gap lock.
5. A bug: the range query on the unique index will access until the first value that does not meet the condition.

Missing update

Loss of updates is another problem caused by locks. Simply put, it means that the update operation of one transaction will be overwritten by the update operation of another transaction, resulting in data inconsistency. E.g:

1. Transaction T1 updates row record r to v1, but transaction T1 does not commit
2. At the same time, transaction T2 updates row record r to v2, transaction T2 is not committed
3. Transaction T1 commit
4. Transaction T2 commit

But under any isolation level of the current database, it will not cause the loss of update problem in the theoretical sense of the database . This is because, for DML operations, you need to lock rows or other granular objects. Therefore, in step 2 above, transaction T2 cannot update row record r, and it will be blocked until transaction T1 is committed.

Although the database can prevent the problem of missing updates at the data level, it cannot prevent the logical loss of updates in production applications, and the problem is not caused by the database itself.

1. Transaction T1 queries a row of data, puts it into local memory, and displays it to the end user User1
2. Transaction T1 also queries the row of data and displays the obtained data to the end user User2
3. User1 modify the row record, update the database and submit
4. User2 modify the row record, update the database and submit

Obviously, in this process, the modification and update operation of User1 is "modified", which may lead to some very scary results. It is also relatively simple to avoid the occurrence of lost updates, just let the transaction become serialized in this case, rather than parallel operation.

MVCC

Under the repeatable read isolation level, the transaction "takes a snapshot" of the database when it is started. This "snapshot" is based on the entire database, and all subsequent additions, deletions, changes, and check operations are performed on this snapshot. In this way, each transaction has its own independent private snapshot database, which is equivalent to operating in its own exclusive snapshot, and naturally the updates of other transactions will not affect me. We call the query using MVCC a consistent read, which not only satisfies repeatable reads, but also solves phantom reads.

You must have found it unrealistic before: if a library has 100G, then I start a transaction and MySQL will copy 100G of data. How slow is this process. However, my affairs are executed very quickly!

With the above foundation, I don't think you will be surprised. In fact, we don't need to copy so much data. Instead, make a statement when the transaction starts: based on the moment I start it, if a data version is generated before I start it, I recognize it; if it is generated after I start it, I don’t recognize it, and I must find it Until the previous version. So we took advantage of the feature of "all data has multiple versions" to achieve the ability to "create snapshots in seconds".

In addition, use begin/start transaction to start a transaction, but it is not the starting point of a transaction. Only after the execution of the first statement operating the InnoDB table after them, the transaction is actually started. Therefore, the creation time of the snapshot also needs to be distinguished. If you want to start a transaction immediately instead of waiting until the first statement is executed, you can use the start transaction with consistent snapshot command;

undo log

The redo log is to restore the submitted data when the database crashes, so the undo log does exactly the opposite of the redo log, which can be understood as our "ctrl+z" undo function. When InnoDB modifies the database, it will not only generate redo log but also undo log.

• For an insert operation, InnoDB will generate a delete;
• For each delete, InnoDB will generate an insert;
• For each update, InnoDB will generate a reverse update and put back the value before modification.

So undo log is a logical log, just for the transaction to fail for some reason, or the user actively executes the rollback statement. Then undo log can "undo" the operations done in the transaction back to the beginning.

The working logic of snapshot in MVCC

We know that undo log can restore the data to the original state of the transaction, so it records every change in the data. The specific form is as follows:

Each transaction in InnoDB has a unique transaction ID called transaction id, which is applied to the InnoDB transaction system at the beginning of the transaction, and it is strictly increasing in the order of application . The transaction id in the above figure has 15, 20, and 25, indicating that the row of data with id=1 has undergone three consecutive updates and has become 3, 5, and 10 different values. The green ellipse is the undo log of each generation of transactions, and v1, v2, and v3 do not actually exist physically, but are calculated according to the current version each time it is needed. For example, when you need v2, you can get the value of v2 through the undo log of v3; when you need v1, v3 pushes back v2, and then v2 pushes back v1 to get it. Therefore, the multi-version of the multi-version concurrency control protocol (MVCC) refers to the undo log marked by the transaction id .

According to the definition of repeatable read, when a transaction is started, you can see the results of all committed transactions. But later, during the execution of this transaction, the updates of other transactions are not visible to it. InnoDB row data has multiple versions, each data version has its own row trx_id, and each transaction or statement has its own consistent view. The common query statement is consistent read, and consistent read will determine the visibility of the data version based on row trx_id and the consistent view.

• For repeatable reads, the query only acknowledges the data that has been submitted before the transaction is started;
• For read submission, the query only acknowledges the data that has been submitted before the statement is started;

The current read always reads the latest version that has been submitted.