background

There are many operations in our actual system, which should produce the same effect or return the same result no matter how many times they are performed. For example:

1. The front end repeatedly submits the selected data, and the background should only generate one response result corresponding to this data.

2. When we initiate a payment request, we should only deduct money from the user account once. When encountering network resend or system bug resend, we should only deduct money once.

3. When sending a message, it should only be sent once. If the same text message is sent to the user, the user will cry.

4. To create a business order, only one business request can be created at a time, and there will be big problems if you create more than one.

In many important situations, these logics need to be supported by idempotent features.

The concept of idempotence

Idempotence (idempotent, idempotence) is a mathematical and computer science concept, common in abstract algebra.

The characteristic of an idempotent operation in programming is that its execution any number of times has the same impact as one execution . An idempotent function, or idempotent method, is a function that can be executed repeatedly with the same arguments and obtain the same result . These functions do not affect the state of the system, and there is no need to worry about changes to the system caused by repeated execution. For example, the "getUsername() and setTrue()" functions are idempotent functions.

The idempotent guarantee of more complex operations is realized by using a unique transaction number (serial number).

My understanding: Idempotence is an operation, no matter how many times it is executed, the effect and the returned result are the same.

Technical solutions

query operation

Query once and query multiple times, in the case of the same data, the query results are the same. select is a natural idempotent operation.

delete operation

The deletion operation is also idempotent, and deleting once and multiple times deletes the data. (Note that the returned results may be different. The deleted data does not exist, and 0 is returned. There are multiple deleted data, and multiple returned results).

unique index

Prevent new dirty data such as: Alipay’s fund account, Alipay also has user accounts, each user can only have one fund account, how to prevent the user from creating multiple fund accounts, then add a unique index to the user ID in the fund account table, So a user successfully adds a fund account record.

Key points: unique index or unique combined index to prevent dirty data in newly added data (when there is a unique index in the table and an error is added during concurrency, just query again, the data should already exist, and just return the result).

pessimistic lock

In the payment scenario, user A's account balance is 150 yuan, and he wants to transfer 100 yuan. Under normal circumstances, user A's balance is only 50 yuan. In general, sql is like this:

update user amount = amount-100 where id=123;

If the same request occurs multiple times, it may cause user A's balance to become negative. In this case, user A may cry. At the same time, system developers may also cry, because this is a very serious system bug.

In order to solve this problem, pessimistic locking can be added to lock the row of data of user A. At the same time, only one request is allowed to obtain the lock and update the data, while other requests wait.

Usually, a single row of data is locked by the following SQL:

select * from user id=123 for update;

The specific process is as follows:

Specific steps:

Multiple requests query user information based on id at the same time.
Determine whether the balance is less than 100, and if the balance is insufficient, return the insufficient balance directly.
If the balance is sufficient, query the user information again through for update and try to acquire the lock.
Only the first request can acquire the row lock, and the rest of the requests that do not acquire the lock wait for the next opportunity to acquire the lock.
After the first request acquires the lock, it is judged whether the balance is less than 100, and if the balance is sufficient, the update operation is performed.
If the balance is insufficient, it means that the request is repeated, and it will directly return success.

Special attention is required: if you are using the mysql database, the storage engine must use innodb, because it only supports transactions. In addition, the id field here must be a primary key or a unique index, otherwise the entire table will be locked.

Pessimistic locking needs to lock a row of data during the same transaction operation. If the transaction takes a long time, it will cause a large number of requests to wait and affect the interface performance.

optimistic lock

Optimistic locking only locks the table at the moment of updating data, and does not lock the table at other times, so it is more efficient than pessimistic locking.

Optimistic locking can be implemented in a variety of ways through version or other status conditions:

(1) Realized by version number

update table_xxx set name=#name#,version=version+1 where version=#version#

(2) Restricted by conditions

update tablexxx set avaiamount=avaiamount-#subAmount# where avaiamount-#subAmount# >= 0

Requirements: quality-#subQuality# >= 0, this scenario is suitable for no version number, only update is for data security verification, suitable for inventory model, deducting shares and rolling back shares, with higher performance..

Note: For the update operation of optimistic lock, it is best to use the primary key or unique index to update. This is a row lock, otherwise the table will be locked when updating. It is better to change the above two sql to the following two

update tablexxx set name=#name#,version=version+1 where id=#id# and version=#version# 

update tablexxx set avaiamount=avaiamount-#subAmount# where id=#id# and avaiamount-#subAmount# >= 0

distributed lock

Still take the example of inserting data. If it is a distributed system, it is difficult to build a global unique index. For example, the unique field cannot be determined. At this time, distributed locks can be introduced. Through a third-party system (redis or zookeeper), in the business The system inserts or updates data, acquires distributed locks, performs operations, and then releases the locks. This is actually the idea of introducing multi-threaded concurrent locks into multiple systems, that is, the solution to distributed systems.

Key points: A long process requires that it cannot be executed concurrently. You can acquire a distributed lock according to a certain flag (user ID + suffix, etc.) One can be executed successfully, and after the execution is completed, the distributed lock is released (the distributed lock must be provided by a third-party system).

select + insert

Background systems with low concurrency, or some task jobs, in order to support idempotence and support repeated execution, the simple processing method is to first query some key data to determine whether it has been executed, and then perform business processing. Note: Do not use this method for core high-concurrency processes.

state machine

In many cases, the business table is stateful. For example, the order table has: 1-order, 2-paid, 3-completed, 4-cancelled and other states. If the values of these states are regular, we can use it to ensure the idempotency of the interface according to the fact that the business nodes are from small to large.

If the order status of id=123 is 已支付, it will become status now 完成.

update `order` set status=3 where id=123 and status=2;

When the first request is made, the status of the order is 已支付, and the value is 2, so the updatestatement can update the data normally. The number of rows affected by the SQL execution result is 1, and the status of the order becomes 3.

When the same request comes later, when the same sql is executed again, since the order status has changed 3, and then used status=2as a condition, the data that needs to be updated cannot be queried, so the number of rows affected by the final sql execution result is, that is 0, it will not be true update data. However, in order to ensure the idempotency of the interface, 0the interface can also directly return success when the number of rows is affected.

The specific flow chart is as follows:

Specific steps:

The user initiates a request through the browser, and the server collects data.
Update to the next state based on the id and the current state as conditions
Determine the number of rows affected by the operation. If one row is affected, the current operation is successful and other data operations can be performed.
If 0 rows are affected, it means that the request is repeated, and success is returned directly.

The main thing to pay special attention to is that this solution is limited to the special case of updating 表有状态字段and just updating , not all scenarios are applicable.状态字段

How to ensure idempotence for APIs that provide external interfaces? For example, the payment interface provided by UnionPay: when merchants need to be connected to submit payment requests, it is attached: source source, seq serial number source+seq is a unique index in the database to prevent multiple payments, ( when concurrent , can only handle one request)

Key point: In order to support idempotent calls in the external interface, the interface has two fields that must be passed, one is the source, and the other is the sequence number seq of the source. These two fields are used as a joint unique index in the provider system , so that when the first When the third party calls, first check in the local system to see if it has been processed, and return the corresponding processing result; if it has not been processed, perform corresponding processing, and return the result. Note that in order to be idempotent and friendly, you must first check whether the business has been processed. If you do not query and directly insert into the business system, an error will be reported, but it has actually been processed.

Summarize

Idempotency should be a gene of qualified programmers. It is the primary consideration when designing a system, especially in systems like Alipay, banks, and Internet finance companies that involve money. Accurate, so there can be no problems such as over-deduction and over-payment, which will be difficult to deal with and the user experience will not be good.

How to achieve the idempotency of the interface