foreword
Idempotency is something that developers must consider in their daily development, especially in scenarios involving money transactions such as transfers and payments. If idempotency problems arise, the consequences will be very serious.
This article will share what idempotence is and how to ensure it.
What is idempotency
Idempotence (idempotent, idempotence) is a mathematical and computer science concept, common in abstract algebra .
The characteristic of an idempotent operation in programming is that its execution any number of times has the same impact as one execution . An idempotent function, or idempotent method, is a function that can be executed repeatedly with the same arguments and obtain the same result. These functions do not affect the state of the system, and there is no need to worry about changes to the system caused by repeated execution.
Reasons for idempotence
- There is no restriction on the front end, causing users to submit repeatedly
- Use the browser to go back, or press F5 to refresh, or use the history to submit the form repeatedly
- Network fluctuations, causing repeated requests
- Timeout retry, causing the interface to be called repeatedly
- Timing task settings are unreasonable, resulting in repeated data processing
- When using message queues, messages are repeatedly consumed
How to ensure idempotence
1. Front-end processing
- Click the submit button to gray, or add loading
- Page redirection (PRG), PRG mode that is
POST-REDIRECT-GET, when the user submits the form, the重定向到另外一个提交成功页面，而不是停留在原先的表单页面。这样就避免了用户刷新导致重复提交。同时防止了通过浏览器按钮前进/后退导致表单重复提交。
2. Select before insert + unique index conflict
Before saving the data, we need to select whether the data exists. If the data already exists, return failure (the specific operation depends on the business situation), if the data does not exist, perform the insert operation.
However, in a high-concurrency scenario, there may be two requests for select, but no data is found, and then the insert operation is executed , so there will be duplicate data at this time, so in the database, we need to add a unique index to ensure idempotence.
The flow chart is as follows:
This scheme is applicable to interfaces for new operations, such as user registration.
3. Build deduplication table
In some business scenarios, duplicate data is allowed, and duplicate data is only allowed in a certain link of the process. In this case, it is not appropriate to directly add a unique index to the table, so it is necessary to create a deduplication table .
CREATE TABLE `table_name` (
`id` bigint(15) NOT NULL AUTO_INCREMENT COMMENT '自增主键',
`order_id` varchar(100) NOT NULL COMMENT '订单号',
`create_time` datetime DEFAULT NULL COMMENT '创建时间',
PRIMARY KEY (`id`),
UNIQUE KEY `index_order_id` (`order_id`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='去重表';
The flow chart is as follows:
Special attention should be paid to the anti-duplication table and the business table must be in the same database, and the operations must be in the same transaction.
This solution is suitable for insertion scenarios that have a unique identifier in the business. For example, in the payment business, if an order will only be paid once, the order ID can be used as the unique identifier.
4. Use pessimistic locks
Pessimistic locks, as the name suggests, have strong exclusive and exclusive characteristics. It refers to a conservative attitude towards data being modified by the outside world (including other current transactions of the system, as well as transaction processing from external systems ). Therefore, during the entire data processing process, the data is locked. The realization of pessimistic locking often relies on the locking mechanism provided by the database (only the locking mechanism provided by the database layer can truly guarantee the exclusivity of data access , otherwise, even if the locking mechanism is implemented in this system, there is no guarantee that the external system will not modify data).
In the transaction scenario, the user's account balance is 100 yuan, and 50 yuan is transferred out. Under normal circumstances, the user's balance remains 50 yuan.
update account set amount-50 where id = 123;
If there are multiple identical requests at this time, the user's amount may become negative. So at this time, pessimistic locking can be used to lock the user's row data, and only one request is allowed to obtain the lock at the same time, while other requests wait.
select * from account where id = 123 for update;
The flow chart is as follows:
Special attention is required: if you are using the mysql database, the storage engine must use innodb, because it only supports transactions. In addition, the id field here must be a primary key or a unique index, otherwise the entire table will be locked.
Because pessimistic locking needs to lock a row of data in the same transaction, if the transaction is relatively long, it will cause a large number of requests to wait and affect the performance of the interface.
5. Use optimistic locking
Optimistic Locking (Optimistic Locking) Compared with pessimistic locking , the optimistic locking mechanism adopts a more relaxed locking mechanism. In most cases, pessimistic locking is implemented by the locking mechanism of the database to ensure the maximum degree of exclusivity of the operation. But what comes with it is a lot of overhead in database performance, especially for long transactions , such overhead is often unbearable. The optimistic locking mechanism solves this problem to a certain extent. Optimistic locking is mostly implemented based on the data version (Version) recording mechanism. What is a data version? That is to add a version identifier to the data. In the version solution based on the database table, it is generally realized by adding a "version" field to the database table. When reading out the data, read this version number together, and when updating later, add one to this version number. At this time, the version data of the submitted data is compared with the current version information of the corresponding record in the database table. If the version number of the submitted data is equal to the current version number of the database table, it is updated, otherwise it is considered as expired data.
Optimistic locks are mainly operated based on the version identifier (version), that is, each time the query data is operated, the version identifier (version) must be queried first, and then the update operation is performed according to the version identifier (version).
select id,amount,version from account id = 123;
update account set amount=amount-50,version=version+1 where id=123 and version = 1;
When multiple identical requests query information, the version identifiers are the same. When one of the requests completes the update operation, the number of items affected by subsequent requests is 0.
The flow chart is as follows:
6. According to the state machine
In many cases, business processes are stateful. At this time, a state machine can be used to ensure idempotence.
For example, in the order business, there is a status "1-ordered, 2-paid, 3-completed, 4-cancelled". According to the business process, the status is transferred in sequence, so when updating, we need to follow the The state of this time is used to update the state of the next time.
update order_info set status = 3 where id = 123 and status = 2;
The flow chart is as follows:
7. Use distributed locks
The logic of the distributed lock is that each request tries to acquire the lock through the unique ID of the business. If the acquisition is successful, the subsequent business logic operation will be performed. If the acquisition fails, the request will be discarded and returned directly.
Distributed locks are usually implemented based on redis.
The flow chart is as follows:
Distributed locks are controlled by setting the expiration time of redis. If the expiration time is set too short, repeated requests cannot be effectively prevented; if the expiration time is set too long, it will affect the redis storage space and even affect subsequent business operations. Therefore, it is necessary to set a reasonable expiration time according to the specific business situation.
8. Based on the token mechanism
This scenario consists of two request phases:
1. The client requests the server to apply for a token
2. The client carries the token to request again, and the server verifies the token before proceeding.
The flow chart is as follows:
Here is a point to note:
To verify whether the token exists on the server side, use the method of deleting the key, that is, redis.del(key). If the deletion is successful, it means that the verification token is passed;
You cannot use the operation of searching first and then deleting, that is, redis.get(key) first, and then redis.del(key). This method cannot guarantee idempotency under high concurrency.
References
How to achieve high idempotency of interfaces
How to ensure the idempotence of interfaces under concurrency?