Article directory
Six, according to the state machine
Foreword:
The problem of interface idempotence is a public problem that has nothing to do with language for developers. This article shares some very practical ways to solve this kind of problem. Most of them have been implemented by me in the project and can be used as a reference for friends in need.
I don’t know if you have ever encountered these scenarios:
1. Sometimes when we fill in some forms , we accidentally click the save button twice quickly, and two duplicate data are generated in the table, but the IDs are different.
2. In order to solve the interface timeout problem in our projects , we usually introduce a retry mechanism . The first request to the interface timed out, and the requesting party failed to obtain the return result in time (it may have been successful at this time). In order to avoid returning an incorrect result (this situation cannot directly return failure, right?), the request will be Retry several times, which will also produce duplicate data.
3. When mq consumers read messages, they sometimes read duplicate messages , and if they are not handled properly, duplicate data will also be generated. Yes, these are idempotence issues.
Interface idempotence means that the results of one request or multiple requests initiated by the user for the same operation are consistent, and there will be no side effects caused by multiple clicks.
This type of problem occurs mostly in the interface:
Insert operation, in this case, multiple requests may generate duplicate data.
If the update operation simply updates data, such as: update user set status=1 where id=1 , there is no problem. If there are calculations, such as: update user set status=status+1 where id=1 , multiple requests in this case may cause data errors.
So how do we ensure the idempotence of the interface ?
1. Select before inserting
Normally, in the interface for saving data, in order to prevent duplicate data, we usually select the data based on the name or code field before inserting . If the data already exists, the update operation is performed. If it does not exist, the insert operation is performed.
This solution may be the one we usually use most to prevent duplicate data. However, this solution is not suitable for concurrent scenarios . In concurrent scenarios, it must be used in conjunction with other solutions, otherwise duplicate data will also be generated . I mention it here to avoid anyone getting into trouble.
2. Add pessimistic lock
In the payment scenario, user A's account balance is 150 yuan and he wants to transfer 100 yuan. Under normal circumstances, user A's balance is only 50 yuan. Generally, sql is like this:
update user amount = amount-100 where id=123;If the same request occurs multiple times, it may cause user A's balance to become negative. In this case, user A may cry. At the same time, system developers may also cry because this is a very serious system bug.
In order to solve this problem, you can add a pessimistic lock to lock user A's row of data. Only one request is allowed to obtain the lock and update the data at the same time, while other requests wait.
Usually, a single row of data is locked through the following SQL:
select * from user id=123 for update;The specific process is as follows:
Specific steps:
- Multiple requests query user information based on ID at the same time.
- Determine whether the balance is less than 100. If the balance is insufficient, it will directly return insufficient balance.
- If the balance is sufficient, query the user information again through for update and try to acquire the lock.
- Only the first request can obtain the row lock, and the remaining requests that have not obtained the lock will wait for the next opportunity to obtain the lock.
- After the first request obtains the lock, it determines whether the balance is less than 100. If the balance is sufficient, the update operation is performed.
- If the balance is insufficient, it means that the request is repeated and success will be returned directly.
Special attention should be paid to: If you are using a mysql database, the storage engine must use innodb , because it only supports transactions . In addition, the id field here must be the primary key or unique index, otherwise the entire table will be locked.
Pessimistic locking needs to lock a row of data during the same transaction operation . If the transaction takes a long time, it will cause a large number of requests to wait and affect the interface performance . In addition, it is difficult to guarantee that each request interface will have the same return value, so it is not suitable for idempotent design scenarios, but it can be used in anti-replication scenarios. By the way, there is actually a difference between anti-duplication design and idempotent design . The anti-duplication design is mainly to avoid duplicate data and does not have many requirements for interface returns. In addition to avoiding duplicate data, idempotent design also requires that each request returns the same result.
3. Add optimistic locking
Since pessimistic locking has performance issues , in order to improve interface performance, we can use optimistic locking. You need to add a timestamp or version field to the table . Here we take the version field as an example.
Query the data before updating it:
select id,amount,version from user id=123;If the data exists, assuming that the found version is equal to 1 , then use the id and version fields as query conditions to update the data:
update user set amount=amount+100,version=version+1where id=123 and version=1;While updating the data, version+1 is added, and then the number of rows affected by this update operation is determined. If it is greater than 0, it means that the update was successful. If it is equal to 0, it means that the update did not change the data.
Since the first request for version equal to 1 can be successful, the version becomes 2 after the operation is successful . At this time, if concurrent requests come in, execute the same sql again:
update user set amount=amount+100,version=version+1where id=123 and version=1;This update operation will not actually update the data. In the end, the number of rows affected by the SQL execution result is 0 , because the version has become 2 , and the version=1 in where will definitely not meet the conditions. However, in order to ensure the idempotence of the interface, the interface can directly return success. Because the version value has been modified, then the previous request must have been successful once, and subsequent requests will be repeated.
The specific process is as follows:
Specific steps:
- First query user information based on ID, including the version field.
- According to the id and version field values as parameters of the where condition, the user information is updated, and version+1
- Determine the number of rows affected by the operation. If it affects 1 row, it means it is a request and other data operations can be performed.
- If 0 rows are affected, it means that the request is repeated and success will be returned directly.
Fourth, add a unique index
In most cases, in order to prevent the generation of duplicate data, we will add a unique index to the table. This is a very simple and effective solution.
alter table `order` add UNIQUE KEY `un_code` (`code`);After adding a unique index, the first request for data can be successfully inserted. But for the same request later, when inserting data, a Duplicate entry '002' for key 'order.un_code exception will be reported, indicating that there is a conflict in the unique index.
Although throwing an exception has no effect on the data, it will not cause wrong data. But in order to ensure the idempotence of the interface, we need to catch the exception and return success.
If it is a java program, it needs to catch: DuplicateKeyException exception. If the spring framework is used, it also needs to catch: MySQLIntegrityConstraintViolationException exception.
The specific flow chart is as follows:
Specific steps:
- The user initiates a request through the browser, and the server collects data.
- Insert that data into mysql
- Determine whether the execution is successful, and if successful, operate other data (and possibly other business logic). If the execution fails, catch the unique index conflict exception and directly return success.
Five, build anti-heavy table
Sometimes not all scenarios in the table do not allow duplicate data, only certain scenarios do. At this time, it is obviously not appropriate to directly add a unique index to the table.
In response to this situation, we can solve the problem by building a defense table .
The table can only contain two fields: id and unique index . The unique index can be a unique identifier combined with multiple fields such as name, code, etc., for example: susan_0001.
The specific flow chart is as follows:
Specific steps:
- The user initiates a request through the browser, and the server collects data.
- Insert the data into the mysql anti-duplication table
- Determine whether the execution is successful. If successful, perform other MySQL data operations (and possibly other business logic).
- If the execution fails, catch the unique index conflict exception and directly return success.
Special attention should be paid to the following: the anti-duplication table and the business table must be in the same database, and the operations must be in the same transaction.
Six, according to the state machine
Many times the business table has status. For example, the order table has: 1-order, 2-paid, 3-completed, 4-cancelled and other statuses. If the values of these states are regular, and the business nodes happen to be from small to large, we can use it to ensure the idempotence of the interface.
If the order status of id=123 is paid , it will now change to completed status.
update `order` set status=3 where id=123 and status=2;When the first request is made, the status of the order is paid and the value is 2 , so the update statement can update the data normally. The number of rows affected by the SQL execution result is 1 , and the order status becomes 3 .
The same request comes later, and when the same SQL is executed again, because the order status becomes 3 , and status=2 is used as the condition, the data that needs to be updated cannot be queried, so the number of rows affected by the final SQL execution result is 0 . That is, the data will not actually be updated. However, in order to ensure the idempotence of the interface, when the number of affected rows is 0 , the interface can also directly return success.
The specific flow chart is as follows:
Specific steps:
- The user initiates a request through the browser, and the server collects data.
- Update to the next state based on the id and current state as conditions
- Determine the number of rows affected by the operation. If 1 row is affected, the current operation is successful and other data operations can be performed.
- If 0 rows are affected, it means that the request is repeated and success will be returned directly.
The main thing to note is that this solution is limited to the special case where the table to be updated has a status field and the status field just needs to be updated. It is not applicable to all scenarios.
7. Add distributed locks
In fact, adding a unique index or adding anti-duplicate tables introduced earlier is essentially using the distributed lock of the database , which is also a type of distributed lock. But since the performance of database distributed locks is not very good, we can use: redis or zookeeper instead .
Since many companies' distributed configuration centers now use apollo or nacos instead of zookeeper , we use redis as an example to introduce distributed locks.
At present, there are three main ways to realize the distributed lock of redis:
- setNx command
- set command
- Redission framework
Each option has its own pros and cons, and there are too many related articles, so I won’t go into details here.
The specific flow chart is as follows:
Specific steps:
- The user initiates a request through the browser, the server collects data and generates the order number code as the only business field.
- Use the redis set command to set the order code into redis and set the timeout at the same time.
- Determine whether the setting is successful. If the setting is successful, it means that it is the first request, and the data operation is performed.
- If the setting fails, it means that the request is repeated, and success will be returned directly.
Special attention should be paid to the following: Distributed locks must be set with a reasonable expiration time. If it is set too short, repeated requests cannot be effectively prevented. If the setting is too long, redis storage space may be wasted . It needs to be determined according to the actual business situation.
8. Obtain token
In addition to the above schemes, there is one last scheme using tokens . This solution is a bit different from all previous solutions, requiring two requests to complete a business operation.
- First request to obtain token
- The second request carries this token to complete the business operation.
The specific flow chart is as follows:
The first step is to obtain the token.
The second step is to do specific business operations.
Specific steps:
- When a user visits a page, the browser automatically initiates a token request.
- The server generates a token, saves it in redis, and returns it to the browser.
- The token is carried when the user initiates a request through the browser.
- Query whether the token exists in redis. If it does not exist, it means that it is the first request, and then perform subsequent data operations.
- If it exists, it means that the request is repeated, and it will directly return success.
- In redis, the token will be automatically deleted after the expiration time.
The above solution is designed for idempotence.
If it is an anti-replication design, the flow chart needs to be modified:
