This article mainly explains what idempotence is, how to ensure idempotence when idempotent problems occur, and I hope this article will be helpful to readers;
1. Interface idempotence
The so-called idempotence means that calling a method or interface multiple times will not change the business status, and it can ensure that the results of repeated and single calls are consistent;
2. Under what circumstances should idempotence be considered?
In our development, the main operation is CRUD, in which read operations and delete operations are naturally idempotent. Our main concern is the new and update operations;
Usage scenarios for idempotence:
1. Front-end repeated submission
For example, if you add a new product function, click the save button, and the front-end clicks save multiple times in succession, the back-end will receive multiple request interfaces. If idempotence is not done, multiple records will be created repeatedly, and dirty data will appear;
This is what we call the problem of how to prevent repeated submissions on the front end;
2. Users maliciously swipe orders
For example: in the user voting function, users repeatedly submit votes for a user (by calling the interface). This may cause the interface to receive voting information repeatedly submitted by users, which will cause the voting results to be seriously inconsistent with the facts;
3. Interface timeout retry
When we call a third-party interface, the call may fail due to network and other reasons, so we will add a failure retry mechanism to the interface call (Spring can implement the retry mechanism through the @Retryable annotation)
Since retrying may occur, repeated calls to the interface may occur. If idempotence is not done when calling again at this time, dirty data may appear;
4. Repeated consumption of messages
When using MQ message middleware, there is a retry mechanism on both the production side and the consumer side, which means that the same message may be consumed repeatedly.
Scenarios where the test interface is idempotent:
1. Network fluctuations may cause repeated requests;
2. Users repeat operations and may unintentionally place multiple orders during operations;
3. Use failure or timeout retry mechanism;
4. Refresh the page repeatedly
5. Use the browser back button to repeat the previous operation, resulting in repeated submission of the form;
6. Use browser history to submit the form repeatedly;
7. Repeated HTTP requests from the browser;
8. Repeated execution of scheduled tasks;
3. How to ensure interface idempotence
Elementary way:
1. Determine whether the data exists before inserting;
This is the most basic and must be done in development. Before inserting or updating, it will be judged whether it already exists in the current database. If it exists, repeated insertion is not allowed. If it does not exist, it can be inserted.
2. Do some interactive control on the front end
For example, after the user clicks the save button, the button will be grayed out or in the loading process, and cannot be clicked again, or jump to other pages, which can prevent a large part of the front-end from being submitted repeatedly;
How to ensure idempotence under high concurrency?
1. Based on pessimistic locking
2. Based on optimistic locking
3. Based on status code
4. Based on unique index
5. Based on distributed locks
6. Based on token
etc. In actual use, basic distributed locks are more commonly used;
Below we introduce the principles of each method one by one:
1. Based on pessimistic locking
Definition: When you want to modify a piece of data in the database, in order to avoid being modified by others at the same time, the best way is to directly lock the data to prevent concurrency;
In the case of high concurrency, a business will be executed twice. We can achieve this through pessimistic locking, that is, adding the for update field to the SQL query statement.
Note here:
1) For a locked row of data, a certain field must be indexed, otherwise the table will be locked; such as order_no, add an index;
2) Pessimistic lock locks a row of data during the same transaction operation. Pessimistic locking has poor performance, so it is generally not recommended to use pessimistic locking for this purpose;
2. Based on optimistic locking
Definition: Optimistic locking is very optimistic. Every time you go to get the data, you think that others will not modify it, so you will not lock it. However, when updating, you will judge whether others have updated the data during this period. You can use version No. mechanism.
The so-called optimistic locking is to add a version (version number) field to the table.
The idempotence of the update operation is controlled through the version number. The user queries the data to be modified, the system returns the data to the page, and puts the data version number into the hidden field. The user modifies the data, clicks submit, and the version number is One pass is submitted to the backend, and the backend uses the version number as the update condition;
Note: What optimistic locking can guarantee is the idempotence of the update operation. If your update itself is an idempotent operation, or the install operation, you cannot use optimistic locking.
3. Based on status code
Many business tables have status, such as the order table. Generally, an order has 1-order creation , 2-order confirmation , 3-order payment , 4-order completion , 5-order cancellation and other order processes. When we update the order status
In the first request, the order confirmation status was successfully changed to order payment , and the number of rows affected by the SQL execution result was 1.
In the second request, I also want to change the order confirmation status to order payment , but the number of rows affected by the SQL execution result is 0. If it is 0, then we can directly return success. There is no need to perform subsequent business operations to ensure the idempotence of the interface.
4. Based on unique index
Generally speaking, pessimistic locks, optimistic locks, and status codes are used for update operations to achieve idempotence, while unique indexes are used for insert operations to ensure idempotence.
1) When creating an order, the front end first obtains the order number through the interface, and then brings the order number when requesting the back end. A unique index is added to the order number in the order table. If the same order number is inserted, an error will be reported directly.
2) When consuming MQ messages, messageIdthey are unique. We can add a new consumption record table and use messageId as the primary key. If consumed repeatedly, the same messageId will exist, and an error will be reported directly when inserted.
5. Based on distributed locks
The logic of distributed locks to achieve idempotence is that when a request comes in, first try to obtain the distributed lock. If successful, the business logic will be executed. Otherwise, if the acquisition fails, the request will be discarded and the request will be returned directly to success.
In fact, the pessimistic lock introduced earlier uses the distributed lock of the database, which packages multiple operations into one atomic operation to ensure idempotence. However, due to the poor performance of database distributed locks,
We can use: redis or zookeeper to implement distributed locks instead.
Specific steps:
The user initiates a request through the browser, the server collects data and generates the order number code as the only business field.
Use the redis set command to set the order code into redis and set the timeout at the same time.
Determine whether the setting is successful. If the setting is successful, it means that it is the first request, and the data operation is performed.
If the setting fails, it means that the request is repeated, and success will be returned directly.
6. Based on Token
The core idea of the token mechanism is to generate a unique certificate, that is, a token, for each operation. A token has only one execution right at each stage of the operation, and once the execution is successful, the execution result is saved. For repeated requests, the same result is returned. The application of token mechanism is very wide.
For example: Each request holds a ticket. This ticket is one-time use. It is destroyed after being used once and cannot be reused. This token is equivalent to the concept of a ticket. The token is brought with each interface request. The server verifies the token when processing it for the first time. And this token can only be used once. If the user uses the same token If the request is made twice, the second time will not be processed and will be returned directly.
The characteristic of the token solution is that it requires two requests to complete a business operation.
Generally includes two request stages:
1) The client requests 申请获取tokenand the server generates a token and returns it.
2) For the second request 带着这个token, the server verifies the token and completes the business operation.
Specific steps:
When a user accesses a page, the browser automatically initiates a token request.
The server generates the token, saves it in redis, and then returns it to the browser.
When the user initiates a request through the browser, the token is carried.
Query whether the token exists in redis. If it does not exist, it means that it is the first request and subsequent data operations will be performed.
If it exists, it means that it is a repeated request, and success will be returned directly.
In redis, the token will be automatically deleted after the expiration time.
The main token scheme has certain risks. Let’s analyze it:
1. Delete the token first [delete the token first, then execute the business] or delete the token later [execute the business first, then delete the token];
(a) Deleting first may result in the business not being executed and retrying with the previous token. Due to anti-redesign, the request still cannot be executed;
(b) Deleting the token later is a big problem. It may cause the business to be processed successfully, but the service is interrupted, a timeout occurs, the token is not deleted, and others continue to retry, causing the business to be executed twice;
So we'd better design to delete the token first, and if the business call fails, re-obtain the token and request again.
2. Here token acquisition, comparison and deletion must ensure atomicity
After verifying whether the token exists, do not use redis.get(token), then use redis.del(token). This is not an atomic operation and will still cause idempotent problems in high concurrency situations.
The way we can use it directly redis.del(token):