1 Idempotency

In a word, idempotence is an execution operation, no matter how many times it is executed, the effect and the returned result are the same.

2 Why implement idempotence?

Nowadays, with the rapid development of Internet technology, business is becoming more and more complex, and there are more and more scenarios of high concurrency and key data in the system.

In a distributed system, machine downtime and message loss are also problems that need to be paid attention to, and one of the typical problems is idempotence.

Think about it, an interface exposed to the outside world will receive many requests, if idempotency cannot be guaranteed, what will be the consequences?

When WeChat conducts a deduction operation, it should only debit the user once. When encountering network failures or system bugs, if idempotence is not implemented, will you directly complain in "C language" if the deduction is too much?

Of course, some interfaces are naturally idempotent, such as query operations and deletion operations. Some modification of data is a constant, without other operations, it is also idempotent. Modification operations may or may not be idempotent.

SELECT col1 FROM tab1 WHERE col2 = 2UPDATE tab1 SET col1 = 1 WHERE col2 = 2UPDATE tab1 SET col1 = col1 + 1 WHERE col2 = 2

Only the third of these three SQLs is not idempotent.

POST request is not inherently an idempotent operation. Each call will generate new resources in the system. If you want idempotence, you must implement it in your business.

What needs to be avoided is that idempotency and concurrency safety are not the same thing. Even if you keep submitting payments for the same order, if you deduct money more than once, it means that the operation is not idempotent.

However, if multiple orders are paid at the same time, the final deducted amount is not the sum of these multiple amounts, indicating that this operation has concurrency security issues. This is a problem of two dimensions, which should be discussed and resolved separately.

3 How to achieve idempotence?

(1) Database anti-heavy

Using the unique index feature of the data table, when an error is added during concurrency, query again, the data already exists, and the addition of dirty data is avoided. But be careful not to use uuid as an index field, its size and type will cause the index to be very slow.

In common scenarios, such as likes by the blog/Weibo system, when a user likes a Weibo, the user id is bound to the blog post id, and subsequent likes of the blog post by the user cannot be inserted. Another example is the financial account, which can store the user ID by adding a unique index to the account table, even if a user is repeatedly operated, he can only have one account.

(2) token token mechanism

The token mechanism is an idempotent design with the widest application range. Although there are many implementation methods, the core idea is to generate a unique token certificate for each operation, and the server uses this unique certificate to ensure that the same operation will not be performed multiple times.

Specifically, it can be divided into two stages, obtaining token and using token. Obtain a token before each interface request, and then add this token to the header body of the request in the next request, and the backend will verify it. If the verification passes, the token will be deleted, and the token will be judged again in the next request. With the help of redis cache, the flow chart is as follows:

(3) Distributed lock

The database anti-duplication table can be replaced by a distributed lock. Compared with the deduplication table, the concurrency is placed in the cache, which is more efficient. The limitation is that only one request can be completed at a time.

For example, some business processing processes are very long and require that they cannot be executed concurrently. You can obtain a distributed lock according to a certain flag (user ID + suffix, etc.) before the process is executed. When other processes are executed, acquiring locks will fail, that is, the process at the same time Only one can be executed successfully, and after the execution is completed, the distributed lock is released.

4 Advantages and disadvantages of idempotence

advantage:

Business needs

shortcoming:

(1) The client processing logic is simplified, but the server-side control idempotent logic becomes more complicated;

(2) Changing concurrent execution to serial execution reduces execution efficiency.

5 extensions

Distributed auto-increment ID can learn from the Snowflake algorithm. The advantages are high performance, low latency, and time-ordered; the disadvantage is that it requires independent development and deployment.

Its structure is as follows:

The highest bit is the sign bit, which is always 0 and not available.
41-bit time series, accurate to millisecond level, 41-bit length can be used (1L << 41) / (1000L * 60 * 60 * 24 * 365) = 69 years. Another important function of time bits is that they can be sorted according to time. Note that the 41-bit time interval is not the time interval for storing the current time, but the value obtained after storing the difference between the time interval (current time interval - start time interval). The start time interval here is generally our id generator The time to start using is specified by our program.
10-digit machine ID, the length of 10 digits supports the deployment of up to 1024 nodes.
The 12-bit counting serial number, which is a series of self-incrementing ids, can support the same node to generate multiple ID serial numbers in the same millisecond, and the 12-bit counting serial number supports each node to generate 4096 ID serial numbers per millisecond.

It adds up to exactly 64 bits, which is a Long type. This algorithm is very concise, but still a good ID generation strategy.

references:

[1] Analysis and solution of mutual exclusion and idempotency problems in distributed systems

https://zhuanlan.zhihu.com/p/22820761

[2] High concurrency interface idempotence solution

https://blog.csdn.net/u011635492/article/details/81058153

[3] Idempotency problems and solutions

https://blog.csdn.net/qq_32020035/article/details/105448889

[4] Snowflake Algorithm

https://www.cnblogs.com/grasp/p/12309726.html

[5] Talk about idempotent issues in development

https://segmentfault.com/a/1190000018808510