Multi-threaded high-concurrency programming MySQL database processing

Author: Chen Jinjian
personal blog: HTTPS: //jian1098.github.io
CSDN blog: https: //blog.csdn.net/c_jian
Jane book: https: //www.jianshu.com/u/8ba9ac5706b6
Contact: jian1098 @qq.com

Summary

---

​ Many mainstream programming languages ​​now support multithreading or coroutine processing, such as those that naturally support coroutines go语言, php+swooleetc. But multi-threading is different from single-threading. If multi-threading does not process the database, sometimes the single-threaded program runs without problems, but once multi-threading is turned on, a series of problems that did not occur before will occur to the database.

​ So what needs to be done for a multi-threaded and highly concurrent database? The following is PHP+MySQLan example to illustrate, other programming languages ​​are similar.

Database processing

---

The minimum conditions for concurrent database processing must meet the first 6 items below

Affairs

​ Transaction is one of the core steps to ensure data accuracy and consistency in multi-threaded processing, but the premise is that the storage engine of the MySQL database must be InnoDB , because MyISAM does not support transactions, and each SQL language for InnoDB is encapsulated as a transaction by default. Automatic submission, which will affect the speed, so it is best to put multiple SQL languages ​​between begin and commit to form a transaction;

For example, transactions in the php-laravel framework:

try {
    
    
    DB::beginTransaction();
    
    //业务逻辑和数据库操作
    
    DB::commit();
} catch (Exception $e) {
    
    
    DB::rollback();
    echo $e->getMessage();
}

Row lock

​ Row locks are the second core step to ensure data accuracy and consistency in multi-threaded processing. Adding row locks can ensure that the data in this transaction will not be changed by other threads to ensure that the data is accurate. When other threads access this row It will block; it should be noted that the row lock can only lock one row. If the database where condition contains multiple rows of results, an error will be reported

For example, row lock in php-laravel framework transaction:

try {
    
    
    DB::beginTransaction();
    
    //lockForUpdate()表示加行锁进行更新
    $user = User::where('id',1)->lockForUpdate()->first();		//可行
    $user = User::where('id','>',1)->lockForUpdate()->first();	//这是不行的，会报错
    $user->name = 'test';
    $user->save();
    
    DB::commit();
} catch (Exception $e) {
    
    
    DB::rollback();
    echo $e->getMessage();
}

index

InnoDBRow lock is achieved on the index, rather than on a physical lock rows. The subtext is that if the access does not hit the index, the row lock cannot be used, and it will degenerate into a table lock.

Prohibited operation

The following operations cannot be performed in a transaction with row locks

• Increase and decrease

  Many programming language frameworks orm provide self-increment and self-decrement continuous operations, but things are not allowed in row lock transactions, such as

  User::where(['id'=>1])->increment('stock');
  User::where(['id'=>1])->decrement('stock');
  

• The query statement is outside the transaction

  The execution of the data query must be after the transaction starts, otherwise it will not work

Execution order

​ If the data tables of the lockForUpdate operation in your transaction are A, B, and C tables in order, then the order in which you update the tables should also be updated in the order of the A, B, and C tables;

​ If your two concurrent threads with the same logic need to lock the same data table respectively, the order of the locked tables should remain the same. Take a reverse case: the two threads are locked in the order of A, B, C and B, A, C. When the two threads execute at the same time, when thread 1 locks the A table, it needs to lock the B table, but the B table has been threaded 2 lock; thread 2 is also waiting for thread 1 to release the A list, so that two threads waiting for each other will cause deadlock.

connection pool

​ In the case of multiple threads, if the database connection uses the same connection instance, an error may occur. This depends on the programming language framework. Some frameworks have already taken care of it, but most of them need to be configured to take effect. The application of the connection pool greatly reduces the database connection resource occupation and access speed.

Read and write separation (optional)

​ When the writing pressure of the database increases, the concentration of reading and writing on one database will make the database overwhelmed. Read-write separation is the establishment of multiple distributed databases, which are only written on the main database server and only read on the slave database server. Among them, multiple slave database servers can be deployed to read data. Orm of general programming language supports read-write separation configuration.

Sub-library and sub-table (optional)

​ With the increase in system access, the QPS is getting higher and higher, and the database disk capacity is increasing. Generally, the QPS of the database server has the best performance when it is 800-1200. When it exceeds 2000, the SQL will become very slow and very It is easy to be killed by requests, and the large amount of data in a single table will also cause the database to execute SQL very slowly. In order to cope with this scenario, the idea and technology of sub-database and sub-table has been developed. For MySQL, you can consider using distributed table middleware, for example mycat. In addition, if the data in a single table is too large, the query speed will be very slow. If there is not much time requirement, consider building a history table.

Use cache (optional)

​ In order to reduce the access to the database, the cache database can be used for the data with low timeliness to improve the access speed, such as redis and memcacha.

Load balancing (optional)

​ Load balancing, also known as distributed server, if there is a large number of users and one server cannot support the service, multiple servers need to be deployed for load balancing. For example, use Aliyun's load balancing or build nginx reverse proxy by yourself. In addition, the cache server can also be built in a distributed manner

Common errors

Error message:

ERROR : (2006, 'MySQL server has gone away')

Solution:

There is a lock waiting problem in the transaction that causes the database to block, check whether there is any transaction in your transaction that has not been committed, and whether it has been rolled back before return, continue or break

Error message:

[ERROR] Uncaught exception 'Illuminate\Database\QueryException': [0]Packets out of order. Expected 15 received 1. Packet size=80

Solution:

Check whether the row lock statement in the transaction contains multiple rows, multiple rows can only lock one row, that is to say, the field of your query condition must be unique;

Or check whether the database connection pool is enabled, this problem will also occur when multiple threads use the same database connection

Error message:

ERROR 1305(42000) SAVEPOINT trans*** DOES NOT EXIST

Solution:

Some transactions have not been committed or rolled back, check whether there are any transactions in your transaction that have not been committed, whether it has been rolled back before return, continue or break