Talk about idempotency

idempotence

1. The mathematical concept of idempotence

If in a unary operation, x is any number in a certain set, if f(x) = f(f(x)) is satisfied, then the f operation is idempotent.

The absolute value operation abs(a) = abs(abs(a)) is an idempotent function

If in a binary operation, x is any number in a certain set, if f(x,x) = x is satisfied, provided that the two parameters of the f operation are both x, then we call the f operation also idempotent.

Finding a large value function max(x,x) = x is an idempotent function

2. Overview of idempotence

2.1 Analysis of idempotent business scenarios

Does the production environment often have duplicate data? When troubleshooting the problem, the data is normal again. What is the explanation for this? How can this happen, and it's hard to troubleshoot.

Reason: Duplicate data or data inconsistency occurs (assuming that the program business code is fine), most of which are duplicate requests. Duplicate requests refer to the same request being submitted multiple times for some reason. There are several scenarios that lead to this situation: (essentially: multiple requests)

1) In the microservice scenario, calling the interface in our traditional application architecture will either succeed or fail. But under the microservice architecture, there will be a third situation [unknown], which is timeout. If it times out, the microservice framework will retry.
2) Multiple clicks during user interaction. For example: click the button multiple times quickly.
3) MQ message middleware, repeated message consumption.
4) The interface of the third-party platform (such as: payment success callback interface), because the exception will also lead to multiple asynchronous callbacks.
5) Other middleware/application services may also retry according to their own characteristics.

2.2 Interface idempotence

The idempotency of the interface actually means that the interface can be called repeatedly. In the case of multiple calls by the caller, the final result of the interface is consistent . To be more precise: multiple calls have the same impact on the system, that is, the effect on resources is the same, but the return value is allowed to be different.

2.3 Examples of idempotent business scenarios

Scenario 1: Payment Scenario

1. An order creation interface, the first call timed out, and then the caller retried once.
2. When the order was created, we needed to deduct the inventory. At this time, the interface timed out, and the caller retried once.
3. When the order starts to be paid, after the payment request is sent, a deduction operation occurs on the server, the interface response times out, and the caller retries once. 4.
An order status update interface, the caller sends two consecutive messages, One is created and one is paid. But you receive the payment first, and then you receive the created
5. After the payment is completed, a text message needs to be sent. When a machine receives the message sent by the text message, the processing is slow. The message middleware delivers the message to another machine for processing

Scenario 2: One-click triple connection

Xiaopozhan has a one-click three-link function. Long press can motivate the up master. Everyone has only one one-click three-link opportunity for each video. Even if you like a certain video and operate it multiple times, you can only use one button and three consecutive times.

Scenario 3: Statistics DAU/MAU

DAU/MAU, also known as daily activity/monthly activity, is a statistical indicator used to reflect the operation of websites, Internet applications or online games. Therefore, a user who logs in multiple times in a day or month (or reaches a certain active user judgment mechanism multiple times) can only be regarded as an active user and cannot be counted repeatedly.

2.4 CRUD and idempotence

Some interfaces can naturally achieve idempotence, such as the query interface. For the query, one query or multiple queries have no impact on the system, and the results are the same. Other functions, such as: add, update , Deletion must ensure idempotency.
Take the user table as an example

1. Query, select * from user where xxx, will not make any changes to the data, and is idempotent

2、新增，insert into user(userid, name) values(1, ‘a’)

If userid is the only primary key, that is, if the above business is repeated, only one piece of user data will be inserted, which is idempotent

If userid is not a primary key and can be repeated, then the above business will be operated multiple times, and multiple data will be added, which is not idempotent

3. Modify and distinguish between direct assignment and calculated assignment

Direct assignment, update user set point = 20 where userid = 1, no matter how many times it is executed, the point is the same, with idempotency

Calculation and assignment, update user set point = point + 20 where userid = 1, each operation point data is different, not idempotent

4. Delete, delete from user where userid = 1, multiple operations, the result is the same, with idempotence

Therefore, we can conclude that data without unique primary key constraints, and operations that modify calculation assignment data are not idempotent .
The same is extended to request types get, put, post and delete
1. get: just query, safe and idempotent. Just like the database select operation, there are no side effects. Repeatedly the result is the same.
2. put: send data to change the content, idempotent. Just like update, but not incremented.
3.post: Send data, change the type, just like insert, you can also request resources (non-idempotent)
4.delete: Deleting a resource is like database delete, idempotent.

3. Solutions

3.1 token + redis mechanism

The idempotent scheme of token + redis is suitable for most scenarios. The main idea:

token作为请求的唯一性标示
redis作为存储token的数据库
每次请求先去redis查看token是否存在
不存在，将返回结果缓存到redis
存在，直接返回缓存结果
设置缓存有效期

服务端提供发送token的接口，业务调用接口前先获取token,然后调用业务接口请求时，把token携带过去,服务器判断token是否存在redis中，存在表示第一次请求，可以继续执行业务，执行业务完成后，最后需要把redis中的token删除。

3.2 Optimistic locking mechanism

Optimistic locking here solves the modification scenario of calculation assignment type

update user set point = #{point}+ 20, version = #{version}+ 1 where userid=#{userid} and version=#{version}

After adding the version number, the calculation and assignment business is made idempotent

Disadvantage: Before operating the business, you need to check the current version. The version example is as follows:

• Multiple version control

This method is suitable for update scenarios. For example, if we want to update the name of a product, we can add a version number to the update interface to make it idempotent.

boolean updateGoodsName(int id,String newName,int version);

It can be implemented as follows

update goods set name=#{newName},version=#{version} where id=#{id} and version<${version}

• state machine control

This method is suitable for the case of state machine flow, such as the creation and payment of orders, and the payment of orders must be before. At this time, we can use the int type when designing the state field, and pass the value type The size is used to be idempotent. For example, the creation of an order is 0, the payment success is 10, and the payment failure is -1.

When doing state machine updates, we can control it like this

update `order` set status=#{status} where id=#{id} and status<#{status}

3.3 Unique primary key mechanism

The unique primary key mechanism is to generate a globally unique primary key ID based on the operation and content of the business. Before executing the operation, it is judged whether the operation has been executed according to whether the globally unique primary key ID exists. If it does not exist, store the global ID in the storage system, such as database, redis, etc. If it exists, it means that the method has been executed.

From an engineering point of view, using global IDs to be idempotent can exist as a basic microservice for business. Such services are used in many microservices. To complete such functions in each microservice will There is duplication of workload. In addition, to create a highly reliable idempotent service, many issues need to be considered. For example, although a machine first writes the globally unique primary key ID into the storage, it hangs up after writing, which requires the introduction of the globally unique primary key ID. timeout mechanism. Using a globally unique ID is a general solution that can support insert, update, and delete business operations. But this scheme looks beautiful but it is more troublesome to implement.

All in all, this mechanism takes advantage of the unique constraint of the primary key of the database to solve the problem of idempotence in insert scenarios. However, the requirement for the primary key is not an auto-incrementing primary key, so the business needs to generate a globally unique primary key ID to solve it.

In the scenario of sub-database and sub-table, the routing rules must ensure that the same request is placed in the same database and the same table , otherwise the database primary key constraint will not work, because the primary keys of different databases and tables are irrelevant.

Because there are certain requirements for the primary key, this solution is a bit coupled with the business, and the auto-incrementing primary key cannot be used.

3.4 Mechanism to remove duplicate table

This method is suitable for scenarios where there is a unique mark inserted in the business. For example, in the above payment scenario, if an order will only be paid once, the order ID can be used as a unique identifier. At this time, we can build a deduplication table and use the unique identifier as a unique index. When we implement it, we put the creation of payment documents and writing into the deduplication table in one transaction. If it is created repeatedly, the database will A unique constraint exception is thrown and the operation is rolled back.

To put it simply, it is to insert the unique primary key into the deduplication table, and then perform business operations, and they are in the same transaction. This ensures that when the request is repeated, the request fails because the deduplication table has a unique constraint, avoiding the idempotent problem.

It should be noted here that the deduplication table and the business table should be in the same library, so as to ensure that in the same transaction, even if the business operation fails, the data in the deduplication table will be rolled back . This is a good guarantee of data consistency.

This solution is also commonly used. The deduplication table has nothing to do with the business. Many businesses can share the same deduplication table, as long as the unique primary key is planned.

3.5 Ticket Mechanism

Payment scenario: A single payment request, that is, direct payment, no additional database operations are required. At this time, an asynchronous request is initiated to create a unique ticketId, which is a ticket. This ticket can only be used once and then becomes invalid.

Specific steps:

1.异步请求获取门票
2.调用支付，传入门票
3.根据门票ID查询此次操作是否存在，如果存在则表示该操作已经执行过，直接返回结果；如果不存在，支付扣款，保存结果
4.返回结果到客户端
如果步骤4通信失败，用户再次发起请求，那么最终结果还是一样的.