Edited by xl_echo, welcome to reprint, please declare the source of the article. For more IT, programming cases and information, please contact QQ: 1280023003
Undefeated in a hundred battles, Yibu claims to be always victorious, never succumbed to a hundred defeats, and Yineng strives to move forward. - This is the real power! !
This article is reproduced from: https://blog.csdn.net/zzy2011266/article/details/42496895
Preface
In order to ensure the consistency and integrity of data, any database has a locking mechanism. The pros and cons of the locking mechanism should directly consider the concurrent processing capability and performance of a database system, so the implementation of the locking mechanism has become one of the core technologies of various databases. This chapter will conduct a more detailed analysis of the respective locking mechanisms of MyISAM and Innodb, the two most frequently used storage engines in MySQL.
Introduction to MySQL Locking Mechanism The
database locking mechanism is simply a rule designed by the database to ensure the consistency of data and make various shared resources orderly accessed by concurrent access. For any kind of database, there needs to be a corresponding locking mechanism, so MySQL is no exception. Due to the characteristics of its own architecture, MySQL database has a variety of data storage engines. The characteristics of application scenarios for each storage engine are different. In order to meet the needs of their specific application scenarios, the locking mechanism of each storage engine is for The design is optimized for the specific scenarios they face, so the locking mechanism of each storage engine is also quite different.
In general, MySQL storage engines use three types (levels) of locking mechanisms: row-level locking, page-level locking, and table-level locking. Let's first analyze the characteristics of these three types of MySQL locks and their advantages and disadvantages.
Row-level locking (row-level)
The biggest feature of row-level locking is that the granularity of locked objects is very small, and it is also the smallest locking granularity implemented by major database management software. Because the lock granularity is small, the probability of lock resource contention is also the smallest, which can give the application as much concurrent processing capability as possible and improve the overall performance of some high-concurrency application systems.
Although it can have great advantages in concurrent processing capability, row-level locking also brings many disadvantages. Since the granularity of locking resources is very small, more things need to be done each time to acquire and release the lock, and the consumption will naturally be greater. In addition, row-level locking is also the most prone to deadlocks.
Table-level
locking is the opposite of row-level locking. Table-level locking is the most granular locking mechanism among MySQL storage engines. The biggest feature of the locking mechanism is that the implementation logic is very simple, and the negative impact on the system is minimal. So acquiring locks and releasing locks is fast. Since table-level locks lock the entire table at a time, the deadlock problem that plagues us can be well avoided.
Of course, the biggest negative impact of the high locking granularity is that the probability of locking resource contention will be the highest, which will greatly reduce it.
Page-level locking (page-level)
Page-level locking is a unique locking level in MySQL, and it is not too common in other database management software. The characteristic of page-level locking is that the locking granularity is between row-level locking and table-level locking, so the resource overhead required to acquire the lock and the concurrent processing capability it can provide are also between the above two. In addition, page-level locking, like row-level locking, can cause deadlocks.
In the process of implementing resource locking in the database, as the granularity of locking resources decreases, the amount of memory required to lock the same amount of data is increasing, and the implementation algorithm will become more and more complicated. However, as the granularity of locking resources decreases, the possibility of application access requests encountering lock waiting decreases, and the overall concurrency of the system also increases.
In MySQL database, non-transactional storage engines such as MyISAM, Memory, and CSV mainly use table-level locking, while Innodb storage engine and NDBCluster storage engine use row-level locking, and BerkeleyDB storage engine mainly uses page-level locking. lock method.
Such locking mechanism of MySQL is mainly due to its original history. In the beginning, MySQL wanted to design a locking mechanism that was completely independent of various storage engines, and in the early MySQL databases, MySQL's storage engines (MyISAM and Momery) were designed to be based on "any table at the same time only allows access (including reads) by a single thread". However, with the continuous improvement of MySQL and the continuous improvement of the system, when the MySQL 3.23 version was developed, the MySQL developers had to revise the previous assumptions. Because they found that when a thread is reading a table, another thread can insert the table, but only to the end of the data file. This is what we call Concurrent Insert provided by MySQL since version 3.23.
When Concurrent Insert appeared, MySQL developers had to modify the locking implementation in the previous system, but only added support for Concurrent Insert without changing the overall architecture. But soon after, with the introduction of the BerkeleyDB storage engine, the previous locking mechanism encountered greater challenges. Because the BerkeleyDB storage engine does not have the restriction that MyISAM and Memory storage engines only allow a single thread to access a table at the same time, it reduces the granularity of this single-threaded access restriction to a single page, which once again forces MySQL developers to One modification of the implementation of the locking mechanism.
Due to the introduction of the new storage engine, the locking mechanism cannot meet the requirements, making MySQL people realize that it is impossible to implement a completely independent locking implementation mechanism that meets the requirements of various storage engines. If the overall performance of the storage engine is degraded due to the poor implementation of the locking mechanism, it will definitely hit the enthusiasm of the storage engine provider. Therefore, the engineers had to abandon the original design intention and make changes in the lock implementation mechanism, allowing the storage engine to change the lock type passed in by MySQL through the interface and decide how to lock the data by itself.
Table-level locks
MySQL table-level locks are mainly divided into two types, one is read lock and the other is write lock. In MySQL, these two kinds of locks are mainly maintained through four queues: two store the read and write lock information currently being locked, and the other two store the waiting read and write lock information, as follows:
Current read-lock queue (lock->read)
Pending read-lock queue (lock->read_wait)
Current write-lock queue (lock->write)
Pending write-lock queue (lock->write_wait)
The relevant information of all threads currently holding read locks can be found in the currentread-lockqueue, and the information in the queue is stored in sequence according to the time when the lock was acquired. The information that is waiting for the lock resource is stored in the Pendingread-lockqueue, and the other two queues that store the write lock information also store the information according to the same rules as above.
Although for us users, the locks (table locks) displayed by MySQL are only two types: read locks and write locks, but there are as many as 11 lock types in the internal implementation of MySQL, which is determined by an enumeration in the system. (thr_lock_type) definition, each value is described as follows:
| lock type | illustrate |
|---|---|
| IGNORE | Used internally when a lock request occurs, no information is stored in the lock structure and queue |
| UNLOCK | The type of interaction used to release the lock request |
| READ | Ordinary read lock |
| WRITE | Ordinary write lock |
| READ_WITH_SHARED_LOCKS | Used in Innodb, it is generated as follows: SELECT...LOCKINSHAREMODE |
| READ_HIGH_PRIORITY | high priority read lock |
| READ_NO_INSERT | Locking is not allowed for ConcurentInsert |
| WRITE_ALLOW_WRITE | This type is actually that when the storage engine handles the lock itself, mysqld allows other threads to acquire read or write locks, because even if there is a resource conflict, the storage engine itself will know how to deal with it |
| WRITE_ALLOW_READ | This kind of locking occurs when DDL (ALTERTABLE...) is performed on the table, MySQL can allow other threads to obtain read locks, because MySQL achieves this function by rebuilding the entire table and then RENAME, and the original table can still provide reading during the whole process. Serve |
| WRITE_CONCURRENT_INSERT | The locking method used when ConcurentInsert is in progress. When the lock is in progress, any other read lock requests except READ_NO_INSERT will not be blocked |
| WRITE_DELAYED | Lock type when using INSERTDELAYED |
| WRITE_LOW_PRIORITY | Displays the declared low-level locking mode, generated by setting LOW_PRIORITY_UPDAT=1 |
| WRITE_ONLY | When a lock is abnormally interrupted during the operation, the system needs to perform a CLOSETABLE operation, and the lock type that occurs during this process is WRITE_ONLY |
Read lock When
a new client request applies for a read lock resource, it needs to meet two conditions:
1. The resource requested to be locked is not currently write-locked;
2. There is no higher priority write lock waiting in the Pending write-lockqueue;
If the above two conditions are met, the request will be passed immediately, and the relevant information will be stored in the Currentread-lockqueue, and if any of the above two conditions are not met, it will be forced to enter the waiting queue Pending read-lockqueue Waiting for the release of resources.
Write lock
When a client requests a write lock, MySQL first checks whether there is already information locked on the same resource in the Current write-lockqueue.
If there is no Current write-lockqueue, check the Pending write-lockqueue again. If it is found in the Pending write-lockqueue, you also need to enter the waiting queue and suspend your own thread to wait for the lock resource. Conversely, if the Pending write-lockqueue is empty, go to the next step to detect the Current read-lockqueue. If there is a read lock, you also need to enter the Pending write-lockqueue to wait. Of course, the following two special cases may also be encountered:
The requested lock is of type WRITE_DELAYED;
The requested lock type is WRITE_CONCURRENT_INSERT or TL_WRITE_ALLOW_WRITE, and Current readlock is the READ_NO_INSERT lock type.
When these two special cases are encountered, the write lock will be obtained immediately and entered into the Current write-lock queue
If a write lock that locks the same resource already exists in the Current write-lockqueue at the beginning of the first detection, it can only enter the waiting queue and wait for the release of the corresponding resource lock.
The priority rules of read requests and write lock requests in the write waiting queue are mainly determined by the following rules:
In addition to the read lock of READ_HIGH_PRIORITY, the WRITE write lock in the Pending write-lockqueue can block all other read locks;
READ_HIGH_PRIORITY read lock requests can block write locks in all Pending write-lockqueues;
Except for WRITE write locks, any other write locks in the Pending write-lockqueue have lower priority than read locks.
A write lock occurs after the Current write-lockqueue and blocks requests for all other locks except in the following cases:
With the permission of some storage engines, a WRITE_CONCURRENT_INSERT write lock request can be allowed
When the write lock is WRITE_ALLOW_WRITE, all read and write lock requests except WRITE_ONLY are allowed
When the write lock is WRITE_ALLOW_READ, all read lock requests except READ_NO_INSERT are allowed
When the write lock is WRITE_DELAYED, all read lock requests except READ_NO_INSERT are allowed
When the write lock is WRITE_CONCURRENT_INSERT, all read lock requests except READ_NO_INSERT are allowed
With the continuous development of the MySQL storage engine, the locking mechanism provided by MySQL itself can no longer meet the needs. Many storage engines have extended and transformed their own storage engines based on the locking mechanism provided by MySQL.
The MyISAM storage engine can basically be said to be a storage engine that relies the most on the table-level locking implemented by the locking mechanism provided by MySQL. Although the MyISAM storage engine does not add other locking mechanisms to itself, in order to better To support related features, MySQL has carried out corresponding implementation transformation based on the original locking mechanism to support its ConcurrentInsert feature.
Several other storage engines that support transactions, such as Innodb, NDBCluster, and BerkeleyDB storage engines, let MySQL directly hand over the lock processing to the storage engine itself, and only hold WRITE_ALLOW_WRITE type locks in MySQL.
Since the locking mechanism used by the MyISAM storage engine is completely implemented by the table-level locking provided by MySQL, we will use the MyISAM storage engine as an example storage engine to demonstrate some basic features of table-level locking. To make the example more intuitive, I'll demonstrate it using locks shown to tables: write locks of type WRITE_ALLOW_READ
Row-level locking
Row -level locking is not a locking method implemented by MySQL itself, but implemented by other storage engines, such as the well-known Innodb storage engine and MySQL's distributed storage engine NDBCluster, which all implement row-level locking. level lock.
Innodb locking mode and implementation mechanism
Considering that row-level locking is implemented by each storage engine, and the specific implementation is also different, Innodb is the most widely used storage engine among transactional storage engines, so here we mainly analyze it. The locking feature of Innodb.
In general, the locking mechanism of Innodb and Oracle database have many similarities. Innodb's row-level locks are also divided into two types, shared locks and exclusive locks. In order to allow row-level locks and table-level locks to coexist during the implementation of the locking mechanism, Innodb also uses intent locks (table-level locks). concept, there are two types of intentional shared locks and intentional exclusive locks.
When a transaction needs to lock a resource it needs, if it encounters a shared lock that is locking the resource it needs, it can add a shared lock, but it cannot add an exclusive lock. However, if the resource that you need to lock is already occupied by an exclusive lock, you can only wait for the lock to release the resource before you can acquire the locked resource and add your own lock. The role of the intent lock is that when a transaction needs to acquire a resource lock, if the resource it needs is already occupied by an exclusive lock, the transaction can add a suitable intent lock to the table that locks the row. If you need a shared lock yourself, add an intent shared lock on the table. And if you need to add an exclusive lock on a row (or some rows), first add an intention exclusive lock on the table. Multiple intent shared locks can coexist at the same time, but only one intent exclusive lock can exist at the same time. So, it can be said that Innodb's lock mode can actually be divided into four types: shared lock (S), exclusive lock (X), intention shared lock (IS) and intention exclusive lock (IX), we can summarize the above through the following table The coexistence logical relationship of these four types:
| 1 | Shared lock(s) | Exclusive lock (X) | Intent Shared Lock (IS) | Intentional exclusive lock (IX) |
|---|---|---|---|---|
| Shared lock(s) | compatible | conflict | compatible | conflict |
| Exclusive lock (X) | conflict | conflict | conflict | conflict |
| Intent Shared Lock (IS) | compatible | conflict | compatible | compatible |
| Intentional exclusive lock (IX) | conflict | conflict | compatible | compatible |
plus: Supplement what I have read about intentional locks in the textbook. Intentional locks are to say hello in advance, "I want this lock, sir", which reduces the tediousness of comparing databases one by one. When applying for blocking, it should be carried out in a top-down order; when releasing a blocking, it should be carried out in a bottom-up order; the multi-granularity blocking method with intention lock improves the concurrency of the system and reduces the overhead of locking and unlocking
Although the locking mechanism of Innodb and Oracle have many similarities, the implementations of the two are quite different. In general, Oracle locks data through table-level locking information on the transaction slot on the physical block where a row record needs to be locked, while Innodb locks through the first index key pointing to the data record before and at the end Implemented by marking lock information on the space space after an index key. This locking implementation of Innodb is called "NEXT-KEYlocking" (gap lock), because if the query is executed through a range search, it will lock all index key values in the entire range, even if the key value does not exist. .
Gap lock has a fatal weakness, that is, after locking a range of key values, even some non-existing key values will be locked innocently, resulting in no data in the locked key value range can be inserted during locking. In some scenarios this can be a huge performance hit. The explanation given by Innodb is to organize the occurrence of phantom reads, so they choose gap locks to achieve locking.
In addition to the negative performance impact of gap locks on Innodb, there are several other major performance risks in the way of locking through indexes:
When Query cannot use the index, Innodb will give up the use of row-level locking and use table-level locking instead, resulting in a decrease in concurrency performance;
When the index used by Query does not contain all the filter conditions, the data only wanted by the index key used for data retrieval may have some rows and columns that do not belong to the result set of the Query, but will also be locked because the gap lock is locked is a range, not a specific index key;
When Query uses an index to locate data, if the index key used is the same but the data rows accessed are different (the index is only a part of the filter condition), the same will be locked
Innodb Locks and Deadlocks Under Each Transaction Isolation Level
Innodb implements the four transaction isolation levels of ReadUnCommited, ReadCommited, RepeatableRead and Serializable defined in the ISO/ANSISQL92 specification. At the same time, in order to ensure the consistency of data in transactions, multi-version data access is implemented.
As we have already introduced in the first section, row-level locking will definitely bring deadlock problems, and Innodb is no exception. As for the generation process of deadlock, we will not describe it in detail here. In the following locking example, we will show you the generation process of deadlock through a practical example. Here we mainly introduce how to deal with the deadlock after it is detected in Innodb.
In the transaction management and locking mechanism of Innodb, there is a special mechanism for detecting deadlock, which will detect the existence of the deadlock within a short time after the deadlock occurs in the system. When Innodb detects that a deadlock has occurred in the system, Innodb will select the smaller transaction among the two transactions that caused the deadlock to roll back through the corresponding judgment, and let the other larger transaction complete successfully. What is the standard for Innodb to determine the size of a transaction? This problem is also mentioned in the official MySQL manual. In fact, after Innodb finds a deadlock, it will calculate the amount of data inserted, updated or deleted by the two transactions to determine the size of the two transactions. That is to say, the more records that a transaction has changed, the less it will be rolled back in a deadlock. But one thing to note is that when more than Innodb storage engine is involved in the deadlock scenario, Innodb cannot detect the deadlock. At this time, the deadlock can only be solved by locking the timeout limit. . In addition, an example of the deadlock generation process will be demonstrated in the Innodb lock example at the end of this section.
Innodb locking mechanism example
mysql> create table test_innodb_lock (a int(11),b varchar(16)) engine=innodb;
Query OK, 0 rows affected (0.02 sec)
mysql> create index test_innodb_a_ind on test_innodb_lock(a);
Query OK, 0 rows affected (0.05 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> create index test_innodb_lock_b_ind on test_innodb_lock(b);
Query OK, 11 rows affected (0.01 sec)
Records: 11 Duplicates: 0 Warnings: 0Reasonable use of lock mechanism to optimize MySQL
MyISAM table lock optimization suggestion
For the MyISAM storage engine, although the additional cost of using table-level locking in the process of locking implementation is smaller than the additional cost of implementing row-level locking or page-level locking, the locking itself consumes resources are minimal. However, due to the granularity of locking, the contention of lock resources will be more than other lock levels, which will reduce the concurrent processing capacity to a large extent.
Therefore, when optimizing the locking problem of the MyISAM storage engine, the most important thing is how to improve the concurrency. Since the lock level is impossible to change, we first need to make the lock time as short as possible, and then make possible concurrent operations as concurrent as possible.
1. Shorten the lock time
Shortening the locking time, just a few words, sounds easy to say, but it is probably not so simple to do in practice. How to make the lock time as short as possible? The only way is to make our query execution time as short as possible.
Do the best to reduce large and complex queries, and split complex queries into several small query distributions;
Build efficient indexes as much as possible to make data retrieval faster;
Try to let the table of the MyISAM storage engine only store the necessary information and control the field type;
Take advantage of the right opportunity to optimize MyISAM table data files;
2. Separate operations that can be parallelized
When it comes to the MyISAM table lock, and it is a table lock that blocks reading and writing, some people may think that the table in the MyISAM storage engine can only be completely serialized, and there is no way to parallelize it. Don't forget that MyISAM's storage engine also has a very useful feature, which is the ConcurrentInsert feature.
The MyISAM storage engine has a parameter option that controls whether to enable the Concurrent Insert function: concurrent_insert, which can be set to 0, 1 or 2. The three values are specified as follows:
concurrent_insert=2, regardless of whether there is free space left by deleting data in the middle part of the table data file of the MyISAM storage engine, ConcurrentInsert is allowed at the end of the data file;
concurrent_insert=1, when there is no free space in the MyISAM storage engine table data file, ConcurrentInsert can be performed from the end of the file;
concurrent_insert=0, no matter whether there is free space left by deleting data in the middle part of the table data file of the MyISAM storage engine, ConcurrentInsert is not allowed.
3. Rational use of read and write priorities
In the various lock analysis section of this chapter, we learned that MySQL's table-level locking has different priority settings for reading and writing. By default, the write priority is greater than the read priority. So, if we can decide the priority of reading and writing according to the difference of each system environment. If our system is mainly read and we want to give priority to query performance, we can set the priority of writing to be lower than the priority of reading by setting the system parameter option low_priority_updates=1, so that we can tell MySQL to try as much as possible. Read requests are processed first. Of course, if our system needs to guarantee the performance of data writing in a limited way, then we don't need to set the low_priority_updates parameter.
Here we can take advantage of this feature and set the concurrent_insert parameter to 1. Even if the possibility of data being deleted is very small, if the temporary waste of a small amount of space is not particularly concerned, set the concurrent_insert parameter to 2. Can try. Of course, there is empty space in the middle of the data file. When the space is wasted, it will also cause more data to be read during the query. Therefore, if the amount of deletion is not small, it is recommended to set concurrent_insert to 1. More appropriate .
Innodb row lock optimization suggestion
Since the Innodb storage engine implements row-level locking, although the performance loss caused by the implementation of the locking mechanism may be higher than that of table-level locking, it is far superior in terms of overall concurrent processing capability. Table-level locking in MyISAM. When the system concurrency is high, the overall performance of Innodb will have obvious advantages compared with MyISAM. However, Innodb's row-level locking also has its fragile side. When we use it improperly, the overall performance of Innodb may not be higher than that of MyISAM, and may even be worse.
In order to make reasonable use of Innodb's row-level locking and avoid weaknesses, we must do the following:
As far as possible, all data retrieval is done through the index, so as to avoid Innodb being upgraded to a table-level lock because it cannot be locked through the index key;
Reasonable design of the index, so that Innodb can be as accurate as possible when locking on the index key, narrow the scope of the lock as much as possible, and avoid unnecessary locks that affect the execution of other queries;
Reduce the range-based data retrieval filter conditions as much as possible to avoid locking records that should not be locked due to the negative impact of gap locks;
Try to control the size of the transaction and reduce the amount of locked resources and the length of the locked time;
When the business environment allows, try to use a lower level of transaction isolation to reduce the additional cost of MySQL due to the realization of the transaction isolation level;
Due to the row-level locking and transactional nature of Innodb, there will definitely be deadlocks. The following are some of the more commonly used methods to reduce the probability of deadlocks.
Readers and friends can try according to their own business characteristics: a) In similar business modules, try to access in the same access order as possible to prevent deadlocks; b) In the same transaction, as far as possible Lock all the resources needed at one time and reduce the probability of deadlock; c) For the business part that is very prone to deadlock, you can try to upgrade the locking granularity and reduce the probability of deadlock through table-level locking;
System lock contention query For two lock levels, there are two sets of special state variables in MySQL to record the contention of lock resources in the system. Let's take a look first.
Contention state variables for table-level locking implemented by MySQL:
mysql> show status like ‘table%’;
+———————–+——-+ | Variable_name | Value | +———————–+——-+
| Table_locks_immediate | 100 |
| Table_locks_waited | 0 |
+———————–+——-+
There are two status variables to record the situation of MySQL internal table-level locking. The two variables are described as follows:
Table_locks_immediate: The number of times table-level locks are generated;
Table_locks_waited: The number of times waiting for table-level lock contention occurred;
Both status values are recorded from the time the system is started, and the number is incremented by 1 if there is no corresponding event. If the Table_locks_waited status value here is relatively high, it means that the table-level lock contention in the system is serious, and it is necessary to further analyze why there is more lock resource contention.
The row-level locking used by Innodb is recorded in the system through another set of more detailed state variables, as follows:
Innodb's row-level lock status variable records not only the number of lock waits, but also the total lock duration, the average duration per time, and the maximum duration, and a non-cumulative state quantity that shows the number of waits currently waiting for locks. The description of each state quantity is as follows:
Innodb_row_lock_current_waits: the number of currently waiting locks;
Innodb_row_lock_time: The total length of time locked from the system startup to the present;
Innodb_row_lock_time_avg: Average time spent on each wait;
Innodb_row_lock_time_max: The time it takes to wait for the most frequent time from the system startup to the present;
Innodb_row_lock_waits: The total number of times the system has waited since the system was started;
For these five state variables, the most important ones are Innodb_row_lock_time_avg (average waiting time), Innodb_row_lock_waits (total number of waits) and Innodb_row_lock_time (total waiting time). Especially when the number of waiting times is high, and the waiting time is not small, we need to analyze why there are so many waiting in the system, and then start to specify the optimization plan according to the analysis results.
In addition, Innodb not only provides these five system state variables, but also provides other richer real-time state information for our analysis and use. It can be viewed by the following methods:
1. Open the monitor function of Innodb by creating the InnodbMonitor table:
mysql> create table innodb_monitor(a int) engine=innodb;
Query OK, 0 rows affected (0.07 sec)
2. Then use "SHOWINNODBSTATUS" to view the detailed information (because there are too many output contents, it will not be recorded here);
Some readers and friends may ask why you need to create a table called innodb_monitor first? Because creating this table actually tells Innodb that we start to monitor its detailed status, and then Innodb will record the more detailed transaction and locking information into MySQL's errorlog, so that we can use it for further analysis later.
Summary
This chapter begins with an introduction to locking in MySQL Server, analyzes the basic implementation mechanisms of table-level locking and row-level locking, the most widely used locking methods in MySQL, and uses MyISAM and Innodb, two typical storage engines, as example storage engines. The table-level locking and row-level locking used are analyzed and demonstrated in more detail. Then, by analyzing the characteristics of the two locking methods, corresponding optimization suggestions and strategies are given. Finally, I learned about how to obtain the resource contention status of various locks currently in the system in MySQL Server. I hope the content of this chapter can be helpful to readers and friends in understanding the MySQL locking mechanism.