I. Overview
When developing the order system, we often encounter payment problems. After the user purchases the goods, the payment is deducted successfully, but the network is abnormal when the result is returned. At this time, the money has been deducted. If the user clicks the button again, the second process will be performed. The deduction was successful and the result returned was successful. The user checked the balance and found that more money had been deducted, and the transaction record also became two.
In the previous single-application system, we only needed to put data operations into transactions, and roll back immediately when an error occurred. However, network interruptions or abnormal problems may also occur when responding to the client. If it is guaranteed that the data is consistent throughout the entire order life cycle from creation to successful payment of an order and will not change due to exceptions. This mechanism requires the use of idempotence.
2. What is idempotence?
Idempotence is a mathematical and computer concept that is common in abstract algebra.
An idempotent function, or idempotent method, is a function that can be executed repeatedly with the same parameters and obtain the same result. These functions will not affect the system state, and there is no need to worry about changes to the system caused by repeated execution.
Idempotence: Any multiple executions have the same impact on the resource itself as one execution
The idempotence of the interface actually means that the interface can be called repeatedly. When the caller calls multiple times, the final result of the interface is the same. Some interfaces can be naturally idempotent, such as the query interface, calling once and calling multiple times. times, the results obtained by the system are the same. The results of the query will not change based on the number of calls.
The insert (INSERT) and modify (UPDATE) methods are non-idempotent and need to be processed in the required scenarios through mechanisms to ensure that multiple executions have no side effects.
If delete is executed once or multiple times, the result will be empty (that is, the result is consistent) and has no side effects. Therefore, deletion based on the primary key ID can be considered (pseudo) idempotent. If deleted based on a non-primary key, it will be executed multiple times. There are no side effects (the data is deleted), and it can also be considered to be (pseudo) idempotent.
3. Idempotence requires attention to several key points
- Idempotence is not just that one (or multiple) requests have no side effects on resources (such as querying database operations, there are no additions, deletions, or modifications, so there is no impact on the database)
- Idempotence also includes side effects on the resource during the first request, but subsequent requests will not have side effects on the resource.
- Idempotence focuses on whether subsequent multiple requests have side effects on resources, but does not focus on the results.
- Issues such as network timeout are not within the scope of idempotent discussion.
Idempotence is a promise (rather than an implementation) made by the system service to the outside world. It promises that as long as the interface is called successfully, multiple external calls will have the same impact on the system. A service declared as idempotent will consider failure of external calls as normal, and there will definitely be retries after failure.
4. What is the use of idempotence?
In business development, we often encounter repeated submissions, whether it is due to network problems that cannot receive the request results and re-initiate the request, or the front-end operation jitters cause repeated submissions. In the trading system and payment system, the problems caused by repeated submissions are particularly obvious, such as:
- If the user clicks multiple times on the APP to submit an order, only one order should be generated in the background;
- Initiate a payment request to Alipay. Due to network problems or system bugs, Alipay should only deduct money once. Obviously, services declared idempotent believe that external callers will make multiple calls. In order to prevent multiple external calls from causing multiple changes to the system data state, the service is designed to be idempotent.
- Repeated requests for outbound orders generate multiple outbound order information.
5. Common means used to ensure idempotence
5.1 MVCC solution
Multi-version concurrency control, this strategy mainly uses update with condition (update with condition to prevent) to ensure that multiple external request calls have a consistent impact on the system. In the process of system design, optimistic locking should be used rationally, and other conditions such as version or updateTime (timestamp) should be used to determine the conditions for optimistic locking. This ensures that the update operation will not have too much impact even in concurrent situations. question. For example
select * from tablename where condition=#condition# //取出要跟新的对象，带有版本versoin
update tableName set name=#name#,version=version+1 where version=#version#
Use version during the update process to prevent other operations from concurrently updating the object, resulting in lost updates. In order to avoid failure, a certain retry mechanism is usually required.
5.2 Deduplication table
When inserting data, insert the deduplication table and use the unique index feature of the database to ensure unique logic.
This method is suitable for scenarios where there is a unique target in the business. For example, in the above payment scenario, if an order will only be paid once, the order ID can be used as a unique identifier. At this time, we can build a deduplication table and use the unique identifier as the unique index. When we implement it, we will create the payment document and write it into the deduplication table in one transaction. If it is created repeatedly, the database will If a unique constraint exception is thrown, the operation will be rolled back.
5.3 Deduplication table
select for update, the record corresponding to the order is locked during the entire execution process. Note: Use this as little as possible when DB reading is greater than writing.
5.4 select + insert
For background systems with low concurrency or some task JOBs, in order to support idempotence and repeated execution, the simple processing method is to first query some key data to determine whether it has been executed before proceeding with business processing. Note: Do not use this method for core high-concurrency processes.
5.5 State machine idempotence
When designing document-related business or task-related business, state machines will definitely be involved. That is, there is a state on the business document, and the state will change under different circumstances. Generally, there is a finite state machine. At this time, If the state machine is already in the next state and a change to the previous state occurs at this time, the change cannot be made in theory. In this case, the idempotence of the finite state machine is guaranteed.
This method is suitable when there is a state machine flow, such as the creation and payment of an order. The payment of the order must be before. At this time, we can use the int type when designing the status field, and pass the value type The size is idempotent, for example, the order creation is 1000, and the payment success is 1. Payment failed for 999.
5.6 Token mechanism to prevent repeated submission of pages
Business requirements : The data on the page can only be clicked and submitted once.
Cause : Data is submitted repeatedly due to repeated clicks, network resending, or nginx resending.
Solution :
- Cluster environment: use token plus redis (redis is single-threaded, processing needs to be queued)
- Single JVM environment: use token plus redis or token plus jvm memory
Processing flow :
- Before submitting data, you must apply for a token from the service. The token is placed in redis or jvm memory and the token is valid for
- After submission, the token is verified in the background, the token is deleted, and a new token is generated and returned.
- Token features: To apply, it is valid once, and the current flow can be limited
5.7 How to ensure idempotence of APIs that provide external interfaces
For example, the payment interface provided by WeChat needs to be connected to the merchant to submit a payment request with: source source and seq serial number. source+seq makes a unique index in the database to prevent multiple payments (only one request can be processed during concurrency)
"Summary:" Idempotence should be a gene of qualified programmers. When designing a system, it is the primary consideration, especially in systems such as Alipay, banks, Internet financial companies, etc. that all involve money. It must be efficient. , the data must also be accurate, so problems such as excessive deductions and overpayments cannot occur. This will be difficult to deal with and the user experience will not be good.
5.7 Globally unique ID
If a globally unique ID is used, a global ID is generated based on the operation and content of the business. Before executing the operation, it is judged whether the operation has been executed based on whether the globally unique ID exists. If it does not exist, store the global ID in the storage system, such as database, redis, etc. If it exists, it means that the method has been executed.
From an engineering perspective, using global IDs to be idempotent can exist as a microservice that is the basis of a business. Such services are used in many microservices. If such functions are completed in each microservice, it will There is duplication of workload. In addition, there are many issues that need to be considered to create a highly reliable idempotent service. For example, although a machine writes the global ID to the storage first, it hangs after writing. This requires the introduction of a timeout mechanism for the global ID.
Using a globally unique ID is a general solution that can support insert, update, and delete business operations. However, this solution looks beautiful but is troublesome to implement. The following solution is suitable for specific scenarios, but is relatively simple to implement.
5.8 Distributed lock
When entering the method, first acquire the lock. If the lock is acquired, continue the subsequent process. If the lock is not acquired, wait for the lock to be released until the lock is acquired. When the method is finished executing, the lock is released. Of course, the lock must have a timeout period to prevent the lock from being accidentally released. It is used to solve the idempotence of distributed systems. Commonly used implementation solutions are tools such as redis and zookeeper.
6. Summary
Idempotence adds additional business logic to control idempotence, complicates business functions, changes parallel execution functions to serial execution, and reduces execution efficiency.
Although idempotence complicates business functions and reduces execution efficiency, it is necessary to ensure the correctness of the system. Taking the above example of updating X, on a single server, adding a lock to that code and setting X to volatile will ensure the correctness of the incoming data.
In a distributed environment and X is queried from the database or file, the correctness of the data cannot be guaranteed by the above locking method. At this time, distributed locks need to be used. Therefore, it is very necessary to ensure the idempotence of methods or interfaces, because there cannot be any problems with the data.