Guided reading
This article mainly analyzes the timing of idempotent processing from each layer of the classic system framework from the perspective of R&D personnel, combined with various business scenarios common to R&D personnel. I hope that through the analysis of this article, developers are no longer unfamiliar with idempotent processing in daily development. Grasp the essence that causes requests and interfaces to be idempotent, and avoid falling into this trap again in your work.
The words idempotency and idempotency are all familiar to a researcher. Have you thought deeply about the background of idempotency, why you need idempotency, and how to do it to be idempotent? Today, we will analyze the method of solving idempotency (sex) in combination with the business scenario and the request process.
1 Concept
The concept of idempotency is a mathematical concept, namely: f...(f(f(x))) = f(x). Used in the computer field, it refers to a promise made by an interface or method in the system. The result of repeatedly calling an interface or method on the same resource with the same parameters is the same as the result of calling it once.
2 business scenarios
In terms of business scenarios, for example, in the current ordering service of Internet e-commerce, the same user calls a certain ordering service in a short period of time, and the order can only be successfully placed once; for transfers between bank accounts, account A to account B Transfer, no matter what problem or failure occurs in the system, the transfer can only be successful once; the front-end page submits multiple requests to the back-end for the content of the same form, and the back-end can only give the same result, etc. All belong to the category of idempotence .
Just imagine, if these services provided are not idempotent, when the customer places an order due to unstable network or clicks the order button several times in a row, the actual customer only places one order, and as a result, the system generates more orders for the customer, The platform/merchant will be unbearable. If it is targeted by the "wool party", the loss will be immeasurable; in the transfer between banks, the A account actually only transferred 1 million to the B account, and the B account is charged. To a few million, this is not acceptable in business. Analyzing these business scenarios, the developers found that whether it is an order placing service, a transfer service or a form submission, they are all business requests, and the interfaces or methods that provide these business services should ensure that no matter whether the service is timed out, retried or faulty, etc. In any case, it is necessary to satisfy the business processing result is correct. For one or more requests in the business, the final processing result is consistent, that is, within a certain period of time, the idempotency of the service is actually the idempotency of the request.
3 Architecture Analysis
From the analysis of the system architecture, which layer should idempotent be done, and how?
Figure 1 Classical system frame diagram
The above picture is a most common classic system frame diagram. The web side initiates a request to the backend. At which layer should idempotent be handled? May wish to analyze layer by layer.
Does Nginx need to be idempotent? The main function of Nginx is to be a web server, reverse proxy, load balancing, etc., and forward requests to the back-end server. It does not participate in specific business itself, so Nginx does not need to be idempotent. ;Gateway is responsible for permission verification, security defense, authentication, flow control, protocol conversion, log auditing, monitoring, etc. It does not contain any business processing, so it does not need to do idempotent processing; Service layer usually It is the processing and arrangement of business logic, which may change the data, but for the result of the data change, it still needs to be written to the database through the data access layer, so the Service layer does not need to do data idempotency; the DAO layer is mainly Interact with the database, write the results of the Service layer into the database, and provide the Service layer with the functions of reading and writing to the database.
When writing to the database, different results may be returned for each write. At this time, it needs to be analyzed and treated according to the scenario. The DataBase layer mainly provides data storage and does not participate in specific business logic calculations. Therefore, through the functional analysis of each layer of the architecture, it is concluded that the idempotent processing of requests needs to be processed at the DAO layer to ensure that the results of multiple requests and one request are consistent.
4 Database Operation Analysis
Through the above analysis, it is concluded that idempotency needs to be processed in the DAO layer, and further analysis shows that the operation of the DAO layer is mainly CRUD. The following is an analysis of whether idempotent is required for each operation, and how to do it.
R (read): The corresponding operation SQL statement is select. As long as the query conditions remain unchanged, within a certain period of time, the results returned by executing it once and executing it multiple times must be the same, so it is idempotent and does not need to be processed.
select * from user where id = 1;The result of querying one or more times is consistent, so it is idempotent.
C (create): The corresponding operation SQL statement is insert. At this time, it is necessary to divide the situation. If the primary key of the database used is the database auto-increment, and the anti-duplication of the primary key of the business is not considered, each write to the database is not idempotent. Therefore, in order to ensure idempotency, it is necessary to insert the data before the data is inserted. Do business anti-duplication or add a unique index to the business primary key on the database table. If the database primary key is not self-incrementing, but is written by the business system, it is necessary to do a one-to-one mapping between the database primary key and the business primary key in the business system, or provide the mapping relationship between the database primary key and the business primary key by an independent service to ensure multiple times. The database primary key obtained by the request is consistent with the business primary key, ensuring that the write operation to the database is idempotent. In general, after the same data is written to the database multiple times, can it be guaranteed that there is only one piece of data.
insert into user (id,age,sex,ts) values(1,10,‘male’,2021-07-20 10:22:23);U (update): The corresponding operation SQL statement is update. When updating, you must use absolute values to update, not relative values. Relative value updates may cause the update operation to not be idempotent.
Idempotent:
update user set age = 10 where id = 1;Non-idempotent:
update user set age++ where id = 1;D (delete): The corresponding operation SQL statement is delete. During the delete operation, if the deletion is a range, it is best to prohibit this type of operation in production; the recommended method is to convert the deletion by range operation to query by range first, and then delete by the primary key of the query. And delete by range is not idempotent.
Idempotent:
delete from user where id = 1;Non-idempotent: This type of operation is prohibited.
delete from user where id in （select id from user order by id desc limit 10);5 Common business scenarios
There are many ways to ensure idempotency. Here are some common business scenarios. In practical applications, choose according to business scenarios.
Figure 2 Process flow of page token mechanism
- When the front-end page is submitted, the page token mechanism. When entering the page, obtain the token from the server, store the token on the server side, and bring the token to the server side for verification when submitting; the common processing flow is as follows:
- The optimistic locking mechanism uses the version number of the database to implement optimistic locking. When the database is updated, it is judged whether the version number is consistent with the query, and the consistent update is successful, otherwise the update fails;
- select+insert , before writing data, first query whether the data exists, and return it directly if it exists. If it does not exist, write the data to ensure the correctness of the data written to the database; it is often used in some background systems with low concurrency or to prevent the duplication of tasks implement;
- Pessimistic locking mechanism , generally id is the primary key or unique index, only the current record is locked;
- select * from table where id = '1234' for update;
- Deduplication table , every time a business table is written or updated, first query whether there is a record in the deduplication table, and then operate the business table.
- Database unique index , establishes a unique index for business tables to avoid multiple writing of business data;
- State machine , the business state is conditional before changing, and must be updated according to the set state conditions;
In actual development, ensuring the idempotency of the provided interface or service is a basic technical requirement. I hope this analysis will help developers who have not yet understood idempotency.