Two mysql row lock instances

Deadlock case analysis:

    First introduce the business background: the loan system (Remittance) sends the loan plan to the loan plan verification and upload server (Mocky) through http. After receiving the request, Mocky verifies the request. If the verification is passed, the loan plan is executed. Packaged and sent to the asynchronous message notification center (NotifyServer will create multiple threads according to the configuration when the project starts, and then rotate a log table or queue for completion messages, and send the task to remittance through http for the release plan status update), after verification, a status message is directly returned to indicate whether the loan plan is accepted successfully. Because NotifyServer is an asynchronous message notification center, the timeliness of messages cannot be guaranteed, and if NotifyServer hangs, all previously completed loan plans will not be able to update the status in time. Therefore, in the remittance system, an additional time interval is also designed. Just take the initiative to directly query Mocky through http (and the mocky interface of this http request will return the loan plan information synchronously and directly instead of asynchronously pushing it to the NotifyServer). Let's look at the specific details:

1. Remittance accepts the processing of requests sent asynchronously by Mocky through NotifyServer (partly, only the part of the code that will cause deadlocks):

fastDataWithTemperatureHandler.updateData(plan);
this.genericDaoSupport.executeSQL("UPDATE t_remittance_plan_exec_log "
				+ " SET pg_account_name =:pgAccountName, " + " pg_account_no =:pgAccountNo, "
				+ " opposite_receive_date =:oppositeReceiveDate, " + " complete_payment_date =:completePaymentDate, "
				+ " actual_total_amount =:actualTotalAmount, " + " execution_status =:executionStatus, "
				+ " transaction_recipient =:transactionRecipient, " + " execution_remark =:executionRemark, "
				+ " transaction_serial_no =:transactionSerialNo, " + " exec_rsp_no =:execRspNo, "
				+ " last_modified_time =:lastModifiedTime, "
				+ " plan_credit_cash_flow_check_number =:planCreditCashFlowCheckNumber "
				+ "WHERE remittance_plan_uuid =:remittancePlanUuid " + "AND exec_req_no =:execReqNo "
				+ "AND execution_status =:processingStatus", params);

Among them, fastDataWithTemperatureHandler.updateData(plan) finally executes this sql: (The transaction propagation mechanism is set to  PROPAGATION_REQUIRED, so the following sql is actually in the same transaction as the above sql )

String sql = "UPDATE t_remittance_plan " + " SET pg_account_name =:pgAccountName, "
				+ " pg_account_no =:pgAccountNo, " + " complete_payment_date =:completePaymentDate, "
				+ " actual_total_amount =:actualTotalAmount, " + " execution_status =:executionStatus, "
				+ " execution_remark =:executionRemark, " + " transaction_serial_no =:transactionSerialNo, "
				+ " last_modified_time =:lastModifiedTime " + "WHERE remittance_plan_uuid =:remittancePlanUuid "
				+ "AND execution_status =:processingStatus";

Next, let's look at the code for the trapping task to accept the information directly returned by Mocky:

this.genericDaoSupport.executeSQL(
					"UPDATE t_remittance_plan_exec_log "
							+ " SET pg_account_name =:pgAccountName, "
							+ " pg_account_no =:pgAccountNo, "
							+ " opposite_receive_date =:oppositeReceiveDate, "
							+ " complete_payment_date =:completePaymentDate, "
							+ " actual_total_amount =:actualTotalAmount, "
							+ " execution_status =:executionStatus, "
							+ " transaction_recipient =:transactionRecipient, "
							+ " execution_remark =:executionRemark, "
							+ " transaction_serial_no =:channelSequenceNo, "
							+ " exec_rsp_no =:execRspNo, "
							+ " last_modified_time =:lastModifiedTime, "
							+ " plan_credit_cash_flow_check_number =:planCreditCashFlowCheckNumber "
							+ "WHERE remittance_plan_uuid =:remittancePlanUuid "
							+ "AND exec_req_no =:execReqNo "
							+ "AND execution_status =:processingStatus", params);
			if(execReqNo.equals(latestExecReqNo)) {
				this.genericDaoSupport.executeSQL(
						"UPDATE t_remittance_plan "
							+ " SET pg_account_name =:pgAccountName, "
							+ " pg_account_no =:pgAccountNo, "
							+ " complete_payment_date =:completePaymentDate, "
							+ " actual_total_amount =:actualTotalAmount, "
							+ " execution_status =:executionStatus, "
							+ " execution_remark =:executionRemark, "
							+ " transaction_serial_no =:channelSequenceNo, "
							+ " last_modified_time =:lastModifiedTime "
							+ "WHERE remittance_plan_uuid =:remittancePlanUuid "
							+ "AND execution_status =:processingStatus", params);

From here, we can see that the first method updates the schedule table after receiving the callback information from the message notification center, and then updates the log table, while the second method updates the log table and schedule table first. Under normal circumstances, but when the task frequency is increased (scanning the database once in 30s and querying the corresponding records and sending them to Mocky), a deadlock occurs. Use show engine innodb status. The information is as follows:

*** (1) TRANSACTION:
TRANSACTION 55375121, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 6 lock struct(s), heap size 1136, 4 row lock(s), undo log entries 1
MySQL thread id 69147, OS thread handle 140007280789248, query id 945290175 192.168.122.7 root updating
UPDATE t_remittance_plan_exec_log  SET pg_account_name ='测试专户号',  pg_account_no ='600000000001',  opposite_receive_date ='2017-10-26 19:49:15',  complete_payment_date ='2017-10-26 19:49:15',  actual_total_amount =1500.00,  execution_status =2,  transaction_recipient =1,  execution_remark ='测试置成功',  transaction_serial_no =null,  exec_rsp_no ='121014412319903744',  last_modified_time ='2017-10-26 19:49:16.618',  plan_credit_cash_flow_check_number =3 WHERE remittance_plan_uuid ='c1af7440-14e2-494f-bf7e-f8c5b0ae2d4d' AND exec_req_no ='9030800c-7390-4424-9c05-96601ebe16c0' AND execution_status =1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 437 page no 13445 n bits 384 index remittance_plan_uuid of table `yunxin`.`t_remittance_plan_exec_log` trx id 55375121 lock_mode X waiting
Record lock, heap no 97 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 30; hex 63316166373434302d313465322d343934662d626637652d663863356230; asc c1af7440-14e2-494f-bf7e-f8c5b0; (total 36 bytes);
 1: len 8; hex 0000000000098792; asc         ;;

*** (2) TRANSACTION:
TRANSACTION 55375122, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
6 lock struct(s), heap size 1136, 4 row lock(s), undo log entries 1
MySQL thread id 69090, OS thread handle 140007617246976, query id 945290177 192.168.122.7 root updating
UPDATE t_remittance_plan  SET pg_account_name ='测试专户号',  pg_account_no ='600000000001',  complete_payment_date ='2017-10-26 19:49:15',  actual_total_amount =1500.00,  execution_status =2,  execution_remark ='测试置成功',  transaction_serial_no =null,  last_modified_time ='2017-10-26 19:49:16.619' WHERE remittance_plan_uuid ='c1af7440-14e2-494f-bf7e-f8c5b0ae2d4d' AND execution_status =1
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 437 page no 13445 n bits 384 index remittance_plan_uuid of table `yunxin`.`t_remittance_plan_exec_log` trx id 55375122 lock_mode X
Record lock, heap no 97 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 30; hex 63316166373434302d313465322d343934662d626637652d663863356230; asc c1af7440-14e2-494f-bf7e-f8c5b0; (total 36 bytes);
 1: len 8; hex 0000000000098792; asc         ;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 236 page no 26239 n bits 376 index remittance_plan_uuid of table `yunxin`.`t_remittance_plan` trx id 55375122 lock_mode X waiting
Record lock, heap no 309 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 30; hex 63316166373434302d313465322d343934662d626637652d663863356230; asc c1af7440-14e2-494f-bf7e-f8c5b0; (total 36 bytes);
 1: len 8; hex 0000000000098794; asc         ;;

*** WE ROLL BACK TRANSACTION (2)

The cause of the deadlock is very clear. The first transaction is waiting for the X lock of the corresponding record of the log table (at this time, according to the code, it can be known that it has obtained the X lock of the corresponding record of the plan table), and the second transaction has acquired the X lock. The X lock recorded in the
log table is waiting for the X lock recorded in the plan table. These two transactions wait for each other, causing a deadlock. Innodb automatically rolls back the second transaction to ensure the normal progress of the first transaction. This is precisely because in the case of high concurrency, the task actively obtains the results for processing and accepts the NotifyServer request for processing at the same time, and then updates the schedule table and log table according to the uuid of the same loan plan, because the update order of the two transactions is inconsistent. , resulting in obtaining row locks of different tables respectively, and then waiting for the other party to release the row locks to obtain the row locks of another table. Knowing the reason, it is very simple to deal with. It only needs to unify the update order of this table in the two cases and it can be solved perfectly.

Gap lock case study:

Business background: After the remittance system (remittance) accepts an external loan application (remittanceApplication), it will perform a series of logical verification and business verification (these two verifications are split into different systems for business reasons), and then according to the verification The verification information updates the status of the corresponding loan application. If the verification fails, the application status is set as failed, and the NotifyServer is used to call back the external. After one callback is completed, the status is updated once; The loan plan obtained by parsing is sent to the Mocky system, and the Mocky system calls back after the processing is completed. After accepting the request, the remiitance updates the status of the loan application corresponding to the loan plan submitted again according to the situation. All these updates all have application_uuid as one of the conditions, again, in most scenarios, the code works fine, but under the 'high' concurrency conditions of the test (500*1000), after running for a certain period of time, always The lock wait timeout exception appears as expected. This means that some SQL statements in the system have been trying to acquire the lock and failed. Check the status of innodb through show engine innodb status, and it shows that there is a gap lock, because every time the information at that time is saved, it is displayed with the following status information analogy:

RECORD LOCKS space id 0 page no 1665 n bits 72 index `number` of table `learn`.`test` trx id BA91 lock_mode X locks gap before rec insert intention waiting

An interval lock designed by innodb to prevent phantom reads during gap locks (see my last blog post about mysql locks for details). Update operations on all common indexes will result in interval locks. In this example, application_uuid is the common Index, in the case of high concurrency, each update generates a gap lock, so after a period of time, a large number of gap locks with different gaps will be generated, and if a new loan application is generated at this time, you want to insert it into the database at this time. , and the uuid of this loan application happens to be in the gap of a certain gap lock, it will inevitably wait for the gap lock to be released, thus waiting for the timeout. The processing is very simple. When designing a large number of updates, replace the ordinary index in the condition with the unique index.

Summary: When there is a mysql lock problem, you should use show engine innodb status (if the storage engine used is innodb) to view the status information of innodb, which records the lock status, type, etc. and the transaction is currently executing. sql, use this information to accurately locate the code and find a solution.