Four schemes for realizing the idempotence of SpringBoot interface

What is idempotency
What is interface idempotency
Why do you need to implement idempotence
The impact on the system after introducing idempotency
Idempotency of Restful API interface
How to achieve idempotency
- Option 1: Database unique primary key
- Solution 2: Database Optimistic Locking
- Solution 3: Anti-heavy Token Token
- Solution 4: Pass the unique serial number downstream
Example of implementing interface idempotence
- Maven introduces related dependencies
- Configure parameters for connecting to Redis
- Create and verify Token tool class
- Create the Controller class for the test
- Create a SpringBoot startup class
- Write a test class for testing
final summary

System environment:

Java JDK version: 1.8
Spring Boot version: 2.3.4.RELEASE

Example address:

https://github.com/my-dlq/blog-example/tree/master/springboot/springboot-idempotent-token/

1. What is idempotence

Idempotence is a concept in mathematics and computer science. When a meta-operation in mathematics is idempotent, its effect on any element twice will be the same as the result of its action once. In computer programming, an idempotent operation is characterized by the fact that any number of executions has the same impact as one execution.

An idempotent function or method is one that can be executed repeatedly with the same arguments and obtain the same result. These functions do not affect the state of the system, and there is no need to worry about changes to the system caused by repeated execution.

2. What is interface idempotence

In HTTP/1.1, idempotency is defined. It describes that one or multiple requests for a resource should have the same result for the resource itself (except for problems such as network timeouts), that is, the first request has side effects on the resource, but subsequent requests will not be repeated. side effects on resources.

The side effect here is not to destroy the result or produce unexpected results. That is, any number of executions thereof will have the same impact on the resource itself as one execution.

3. Why do we need to achieve idempotence

In general, when the interface is called, the information can be returned normally and will not be submitted repeatedly, but problems may occur when encountering the following situations, such as:

Repeated submission of forms at the front end: When filling out some forms, the user fills in and submits. Many times, due to network fluctuations, the user does not respond to the user's successful submission in time, causing the user to think that the submission has not been successful, and then keep clicking the submit button. This will happen Repeat form requests.
Malicious swiping of orders by users: For example, when implementing the function of user voting, if a user repeatedly submits votes for a user, this will cause the interface to receive the voting information repeatedly submitted by the user, which will make the voting results seriously inconsistent with the facts.
Repeated submission of interface timeout: In many cases, HTTP client tools enable the timeout retry mechanism by default, especially when a third party calls the interface, in order to prevent request failure caused by network fluctuations and timeouts, a retry mechanism will be added, resulting in multiple submissions of a request. Second-rate.
Repeated consumption of messages: When using MQ message middleware, if an error occurs in the message middleware and the consumption information is not submitted in time, repeated consumption will occur.

The biggest advantage of using idempotency is to make the interface guarantee any idempotent operation, avoiding unknown problems caused by the system due to retries.

4. The impact on the system after introducing idempotency

Idempotency is to simplify the logic processing of the client, and can place operations such as repeated submissions, but it increases the logic complexity and cost of the server, mainly as follows:

Changing the function of parallel execution to serial execution reduces the execution efficiency.
Additional business logic for controlling idempotence is added, which complicates business functions;

Therefore, it is necessary to consider the necessity of introducing idempotence when using it. According to the specific analysis of actual business scenarios, except for the special requirements of the business, the interface idempotence introduced is generally not required.

5. The idempotence of the Restful API interface

Among the several HTTP interface methods recommended by the popular Restful , there are idempotent and non-idempotent methods, as follows:

√ Satisfy idempotence
x is not idempotent
- It may or may not be idempotent, depending on the actual business logic

6. How to achieve idempotency

Option 1: Database unique primary key

Program description

The realization of the unique primary key of the database is mainly to use the characteristics of the unique constraint of the primary key in the database. Generally speaking, the unique primary key is more suitable for the idempotency of "insert", which can ensure that only one record with the unique primary key can exist in a table.

When using the unique primary key of the database to complete idempotency, it should be noted that the primary key is generally not the self-incrementing primary key in the database, but the distributed ID as the primary key, so as to ensure the global uniqueness of the ID in a distributed environment sex.

Applicable operations:

insert operation
delete operation

Use restrictions:

A globally unique primary key ID needs to be generated;

Main process:

Main process:

① The client executes the creation request and calls the server interface.
② The server executes the business logic, generates a distributed ID, uses the ID as the primary key of the data to be inserted, and then executes the data insertion operation and runs the corresponding SQL statement.
③ The server inserts the piece of data into the database. If the insertion is successful, it means that the interface has not been called repeatedly. If a duplicate primary key exception is thrown, it means that the record already exists in the database, and an error message is returned to the client.

Solution 2: Database Optimistic Locking

Program description:

The database optimistic locking scheme is generally only applicable to the process of performing "update operations". We can add an additional field in the corresponding data table in advance to serve as the version identifier of the current data. In this way, each time the data in the table of the database is updated, the version identifier is used as a condition, and the value is the value of the version identifier in the data to be updated last time.

Applicable operations:

update operation

Use restrictions:

Additional fields need to be added to the corresponding business table of the database;

Description example:

For example, in the following data table:

In order to prevent repeated updates every time an update is performed, and to make sure that the update must be the content to be updated, we usually add a version field to record the current record version, so that this value is brought when updating, then as long as the update operation is performed It is determined that the information under a corresponding version must be updated. In this way, every time an update is performed, the version number to be updated must be specified, and the information of version=5 can be accurately updated by the following operations:

UPDATE my_table SET price=price+50,version=version+1 WHERE id=1 AND version=5

After the above WHERE is followed by the condition id=1 AND version=5 is executed, the version of id=1 is updated to 6, so if the SQL statement is executed repeatedly, it will not take effect, because the data of id=1 AND version=5 is no longer valid. exists, so that the idempotency of the update can be kept, and multiple updates will not affect the result.

Solution 3: Anti-heavy Token Token

Program description:

For situations such as continuous clicking by the client or timeout retry by the caller, such as submitting an order, this kind of operation can use the Token mechanism to prevent repeated submissions.

To put it simply, the caller first requests a global ID (Token) from the backend when calling the interface, and carries this global ID together with the request (the Token is best placed in the Headers), and the backend needs to verify the Token As a Key, user information is sent to Redis as a Value for key-value content verification. If the Key exists and the Value matches, the delete command is executed, and then the subsequent business logic is normally executed. If there is no corresponding Key or the Value does not match, an error message will be returned for repeated execution, so as to ensure idempotent operation.

Applicable operations:

insert operation
update operation
delete operation

Use restrictions:

A globally unique Token string needs to be generated;
Need to use the third-party component Redis for data validation;

Main process:

① The server provides an interface to obtain a Token, which can be a serial number, or a distributed ID or UUID string.
② The client calls the interface to obtain the Token, and the server will generate a Token string at this time.
③ Then store the string in the Redis database, and use the Token as the key of Redis (pay attention to setting the expiration time).
④ Return the Token to the client, and after the client gets it, it should be stored in the hidden field of the form.
⑤ When the client submits the form, it stores the Token in the Headers, and executes the business request with the Headers.
⑥ After receiving the request, the server gets the Token from the Headers, and then checks whether the key exists in Redis based on the Token.
⑦ The server judges according to whether the key exists in Redis, and if it exists, deletes the key, and then executes the business logic normally. If it does not exist, an exception is thrown, and an error message for repeated submissions is returned.

Note that in the case of concurrency, the execution of Redis to find data and delete needs to ensure atomicity, otherwise it is very likely that idempotence cannot be guaranteed under concurrency. Its implementation method can use distributed locks or use Lua expressions to cancel query and delete operations.

Solution 4: Pass the unique serial number downstream

Program description:

The so-called request serial number is actually a unique and unique serial number in a short period of time attached to each request to the server. The serial number can be an ordered ID or an order number, which is generally generated by the downstream. The serial number and the ID used for authentication are appended to the upstream server interface.

When the upstream server receives the request information, it combines the serial number with the downstream authentication ID to form a Key for operating Redis, and then checks in Redis whether there is a key-value pair for the corresponding Key. According to the result:

If it exists, it means that the downstream request for the serial number has been processed. At this time, the error message of the repeated request can be directly responded.
If it does not exist, use the Key as the key of Redis, use the downstream key information as the stored value (such as some business logic information passed by the downstream business), store the key-value pair in Redis, and then execute the corresponding business normally Just logic.

Applicable operations:

insert operation
update operation
delete operation

Use restrictions:

Requiring third parties to deliver unique serial numbers;
Need to use the third-party component Redis for data validation;

Main process:

The main steps:

① The downstream service generates the distributed ID as the serial number, and then executes the request to call the upstream interface, with the "unique serial number" and the requested "authentication credential ID".
② The upstream service performs security verification to detect whether there are "serial number" and "credential ID" in the parameters passed downstream.
③ The upstream service goes to Redis to check whether there is a corresponding Key composed of "serial number" and "authentication ID". If it exists, it throws an exception message for repeated execution, and then responds to the corresponding error message downstream. If it does not exist, use the combination of "serial number" and "authentication ID" as the Key, and use the downstream key information as the Value, and then store it in Redis, and then execute the incoming business logic normally.

When inserting data into Redis in the above steps, the expiration time must be set. This can ensure that within this time range, if the interface is called repeatedly, it can be judged and identified. If the expiration time is not set, it is likely to cause an unlimited amount of data to be stored in Redis, resulting in Redis not working properly.

7. Example of implementing interface idempotence

The anti-heavy Token token scheme is used here, which can guarantee idempotency under different request actions. The implementation logic can be seen in the "anti-heavy Token token" scheme written above, and then write down the code to realize this logic.

1. Maven introduces related dependencies

Here, Maven tools are used to manage dependencies. Here, SpringBoot, Redis, and lombok related dependencies are introduced in pom.xml.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.4.RELEASE</version>
    </parent>

    <groupId>mydlq.club</groupId>
    <artifactId>springboot-idempotent-token</artifactId>
    <version>0.0.1</version>
    <name>springboot-idempotent-token</name>
    <description>Idempotent Demo</description>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <!--springboot web-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <!--springboot data redis-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-pool2</artifactId>
        </dependency>
        <!--lombok-->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

2. Configure the parameters for connecting to Redis

Configure the parameters for connecting to Redis in the application configuration file, as follows:

spring:
  redis:
    ssl: false
    host: 127.0.0.1
    port: 6379
    database: 0
    timeout: 1000
    password:
    lettuce:
      pool:
        max-active: 100
        max-wait: -1
        min-idle: 0
        max-idle: 20

3. Create and verify Token tool class

Create a Service class for operating Token, which contains Token creation and verification methods, among which:

Token creation method: use the UUID tool to create a Token string, set "idempotent_token:"+"Token string" as the Key, use user information as the Value, and store the information in Redis.
Token verification method: Receive the Token string parameter, add the Key prefix to form the Key, then pass in the value value, and execute the Lua expression (Lua expression can guarantee the atomicity of command execution) to find the corresponding Key and delete the operation. After the execution is completed, verify the return result of the command. If the result is not empty and non-zero, the verification is successful, otherwise it fails.

import java.util.Arrays;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.stereotype.Service;

@Slf4j
@Service
public class TokenUtilService {

    @Autowired
    private StringRedisTemplate redisTemplate;

    /**
     * 存入 Redis 的 Token 键的前缀
     */
    private static final String IDEMPOTENT_TOKEN_PREFIX = "idempotent_token:";

    /**
     * 创建 Token 存入 Redis，并返回该 Token
     *
     * @param value 用于辅助验证的 value 值
     * @return 生成的 Token 串
     */
    public String generateToken(String value) {
        // 实例化生成 ID 工具对象
        String token = UUID.randomUUID().toString();
        // 设置存入 Redis 的 Key
        String key = IDEMPOTENT_TOKEN_PREFIX + token;
        // 存储 Token 到 Redis，且设置过期时间为5分钟
        redisTemplate.opsForValue().set(key, value, 5, TimeUnit.MINUTES);
        // 返回 Token
        return token;
    }

    /**
     * 验证 Token 正确性
     *
     * @param token token 字符串
     * @param value value 存储在Redis中的辅助验证信息
     * @return 验证结果
     */
    public boolean validToken(String token, String value) {
        // 设置 Lua 脚本，其中 KEYS[1] 是 key，KEYS[2] 是 value
        String script = "if redis.call('get', KEYS[1]) == KEYS[2] then return redis.call('del', KEYS[1]) else return 0 end";
        RedisScript<Long> redisScript = new DefaultRedisScript<>(script, Long.class);
        // 根据 Key 前缀拼接 Key
        String key = IDEMPOTENT_TOKEN_PREFIX + token;
        // 执行 Lua 脚本
        Long result = redisTemplate.execute(redisScript, Arrays.asList(key, value));
        // 根据返回结果判断是否成功成功匹配并删除 Redis 键值对，若果结果不为空和0，则验证通过
        if (result != null && result != 0L) {
            log.info("验证 token={},key={},value={} 成功", token, key, value);
            return true;
        }
        log.info("验证 token={},key={},value={} 失败", token, key, value);
        return false;
    }

}

4. Create the Controller class for the test

Create a Controller class for testing, which contains an interface for obtaining Token and testing the idempotency of the interface, the content is as follows:

import lombok.extern.slf4j.Slf4j;
import mydlq.club.example.service.TokenUtilService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

@Slf4j
@RestController
public class TokenController {

    @Autowired
    private TokenUtilService tokenService;

    /**
     * 获取 Token 接口
     *
     * @return Token 串
     */
    @GetMapping("/token")
    public String getToken() {
        // 获取用户信息（这里使用模拟数据）
        // 注：这里存储该内容只是举例，其作用为辅助验证，使其验证逻辑更安全，如这里存储用户信息，其目的为:
        // - 1)、使用"token"验证 Redis 中是否存在对应的 Key
        // - 2)、使用"用户信息"验证 Redis 的 Value 是否匹配。
        String userInfo = "mydlq";
        // 获取 Token 字符串，并返回
        return tokenService.generateToken(userInfo);
    }

    /**
     * 接口幂等性测试接口
     *
     * @param token 幂等 Token 串
     * @return 执行结果
     */
    @PostMapping("/test")
    public String test(@RequestHeader(value = "token") String token) {
        // 获取用户信息（这里使用模拟数据）
        String userInfo = "mydlq";
        // 根据 Token 和与用户相关的信息到 Redis 验证是否存在对应的信息
        boolean result = tokenService.validToken(token, userInfo);
        // 根据验证结果响应不同信息
        return result ? "正常调用" : "重复调用";
    }

}

5. Create a SpringBoot startup class

Create a startup class for starting SpringBoot applications.

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}

6. Write a test class for testing

Write a test class to test, access the same interface multiple times, and test whether it can be executed successfully only the first time.

import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.http.MediaType;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.request.MockMvcRequestBuilders;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import org.springframework.web.context.WebApplicationContext;

@Slf4j
@SpringBootTest
@RunWith(SpringRunner.class)
public class IdempotenceTest {

    @Autowired
    private WebApplicationContext webApplicationContext;

    @Test
    public void interfaceIdempotenceTest() throws Exception {
        // 初始化 MockMvc
        MockMvc mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext).build();
        // 调用获取 Token 接口
        String token = mockMvc.perform(MockMvcRequestBuilders.get("/token")
                .accept(MediaType.TEXT_HTML))
                .andReturn()
                .getResponse().getContentAsString();
        log.info("获取的 Token 串：{}", token);
        // 循环调用 5 次进行测试
        for (int i = 1; i <= 5; i++) {
            log.info("第{}次调用测试接口", i);
            // 调用验证接口并打印结果
            String result = mockMvc.perform(MockMvcRequestBuilders.post("/test")
                    .header("token", token)
                    .accept(MediaType.TEXT_HTML))
                    .andReturn().getResponse().getContentAsString();
            log.info(result);
            // 结果断言
            if (i == 0) {
                Assert.assertEquals(result, "正常调用");
            } else {
                Assert.assertEquals(result, "重复调用");
            }
        }
    }

}

Displayed as follows:

[main] IdempotenceTest:  获取的 Token 串：980ea707-ce2e-456e-a059-0a03332110b4
[main] IdempotenceTest:  第1次调用测试接口
[main] IdempotenceTest:  正常调用
[main] IdempotenceTest:  第2次调用测试接口
[main] IdempotenceTest:  重复调用
[main] IdempotenceTest:  第3次调用测试接口
[main] IdempotenceTest:  重复调用
[main] IdempotenceTest:  第4次调用测试接口
[main] IdempotenceTest:  重复调用
[main] IdempotenceTest:  第5次调用测试接口
[main] IdempotenceTest:  重复调用

8. Final summary

Idempotency is a very common and important requirement in development, especially for services linked to money such as payments and orders. It is especially important to ensure interface idempotence. In actual development, we need to flexibly choose idempotent implementation methods for different business scenarios:

For orders and other unique primary keys, you can use the "unique primary key scheme" to achieve.
For updating scene operations related to updating order status, it is easier to use the "optimistic locking scheme".
For the upstream and downstream, it is more reasonable for the downstream to request the upstream, and the upstream service can use the "downstream transfer unique serial number scheme".
Similar to the scenario where the front-end repeatedly submits, orders repeatedly, and does not have a unique ID number, it can be realized more quickly through the " anti-duplication Token solution " that combines Token and Redis.

The above is just to give some suggestions. Let me emphasize again that to realize idempotence, you need to understand your own business needs first, and it is reasonable to realize it according to the business logic. Only by dealing with the details of each node and improving the overall business process design can it be more A good guarantee for the normal operation of the system. Finally, make a brief summary, and then this blog post ends here, as follows: