[Spring Boot] 4 ways to achieve interface idempotence

[Spring Boot] 4 ways to achieve interface idempotence

1. What is idempotence?

Idempotence is a concept in mathematics and computer science. In mathematics, when a certain element operation is idempotent, its effect on any element twice will have the same result as its effect once. In computer programming, the characteristic of an idempotent operation is that any number of its executions have the same impact as one execution . An idempotent function or idempotent method is a function that can be executed repeatedly with the same parameters and obtain the same result. These functions will not affect the system state, and there is no need to worry about changes to the system caused by repeated execution.

2. What is interface idempotence?

In HTTP/1.1, idempotence is defined. It describes that one and multiple requests for a resource should have the same result for the resource itself (except for problems such as network timeout), that is, the first request has side effects on the resource, but subsequent requests will not cause any side effects. Have side effects on resources . The side effects here do not damage the results or produce unpredictable results. In other words, any multiple executions have the same impact on the resource itself as one execution.

3. Why we need to achieve idempotence

Under normal circumstances, when the interface is called, the information can be returned normally and will not be submitted repeatedly. However, problems may occur when encountering the following situations, such as:

• Repeated submission of forms on the front end: When filling in some forms, the user completes the submission. In many cases, due to network fluctuations, the user does not receive a successful submission response in time, causing the user to think that the submission was not successful, and then keeps clicking the submit button. This will happen. Submit form requests repeatedly;
• Users maliciously commit fraud: For example, when implementing a user voting function, if a user repeatedly submits a vote for a user, this will cause the interface to receive voting information repeatedly submitted by the user, which will cause the voting results to be seriously inconsistent with the facts;
• Interface timeout and repeated submission: Many HTTP client tools enable the timeout retry mechanism by default, especially when a third party calls the interface. In order to prevent request failures caused by network fluctuations, timeouts, etc., a retry mechanism will be added, resulting in multiple submissions of one request. Second-rate;
• Repeated consumption of messages: When using MQ message middleware, if an error occurs in the message middleware and consumption information is not submitted in time, repeated consumption will occur.

The biggest advantage of using idempotence is that the interface guarantees any idempotent operation and avoids unknown problems caused by retries, etc. in the system.

4. The impact on the system after introducing idempotence

Idempotence is to simplify client logic processing and can place operations such as repeated submissions, but it increases the logic complexity and cost of the server. The main ones are:

• Change the parallel execution function to serial execution, which reduces the execution efficiency;
• Additional business logic to control idempotence is added, complicating business functions.

Therefore, you need to consider whether to introduce idempotence when using it. According to the specific analysis of the actual business scenario, except for special business requirements, generally there is no need to introduce interface idempotence.

5. Idempotence of Restful API interface

Among the several HTTP interface methods recommended by popular Restful, there are idempotent lines and methods that cannot guarantee idempotence, as follows:

• √ Satisfies idempotence
• x is not idempotent
• - It may or may not satisfy idempotence, depending on the actual business logic

method type	Is it idempotent?	describe
Get	√	Get method is used to obtain resources. It generally does not and should not change system resources, so it is idempotent.
Post	×	The Post method is generally used to create new resources. Each time it is executed, new data will be added, so it is not idempotent.
Put	-	The Put method is generally used to modify resources. This operation depends on the situation to determine whether it is idempotent. In the update operation, updating directly based on a certain value can also maintain idempotence. However, updates that perform accumulation operations are not idempotent.
Delete	-	The Delete method is generally used to delete resources. This operation depends on the situation to determine whether it is idempotent. When deleting based on a unique value, deleting the same data multiple times will have the same effect. However, it should be noted that deletion with query conditions does not necessarily satisfy idempotence. For example, after deleting a batch of data according to a condition, if a new piece of data is added that also satisfies the condition, and then another deletion is performed, the newly added data that satisfies the condition will also be deleted.

6. How to achieve idempotence

Database unique primary key

Solution description

The implementation of the unique primary key in the database mainly takes advantage of the unique constraint characteristics of the primary key in the database. Generally speaking, the unique primary key is more suitable for idempotence during "insertion", which can ensure that only one record with the unique primary key can exist in a table. When using the database's unique primary key to achieve idempotence, it should be noted that the primary key generally does not use the auto-incremented primary key in the database, but uses the distributed ID as the primary key, so as to ensure the global uniqueness of the ID in a distributed environment . sex .

Applicable operations:

• insert operation
• Delete operation

Usage restrictions:

• A globally unique primary key ID needs to be generated;

Main process:

Main process:

• The client executes the creation request and calls the server interface;
• The server executes the business logic, generates a distributed ID, uses the ID as the primary key of the data to be inserted, then performs the data insertion operation and runs the corresponding SQL statement;
• The server inserts the piece of data into the database. If the insertion is successful, it means that the interface is not called repeatedly. If a primary key duplication exception is thrown, it means that the record already exists in the database, and an error message will be returned to the client.

Database optimistic locking

Program description:

The database optimistic locking scheme is generally only applicable to the process of performing "update operations". We can add an additional field to the corresponding data table in advance to serve as the version identifier of the current data. In this way, every time this piece of data in this table of the database is updated, the version identifier will be used as a condition, and the value is the value of the version identifier in the last data to be updated.

Applicable operations:

• update operation

Usage restrictions:

• Additional fields need to be added to the corresponding business table in the database;

Description example:

For example, the following data table exists:

pid	name	price
1	Xiaomi mobile phone	1000
2	iPhone	2500
3	Huawei cell phone	1600

In order to prevent repeated updates every time an update is performed, and to ensure that the updated content must be the content to be updated, we usually add a version field to record the current record version, so that the value is brought when updating, and then as long as the update operation is performed Make sure that the information that must be updated is the information under a corresponding version.

pid	name	price	version
1	Xiaomi mobile phone	1000	10
2	iPhone	2500	21
3	Huawei cell phone	1600	5

In this way, every time you perform an update, you must specify the version number to be updated. The following operations can accurately update the version=5 information:

UPDATE my_table SET price = price + 50, version = version + 1 WHERE pid = 1 AND version = 5

The above WHERE is followed by the condition pid = 1 AND version = 5. After being executed, the version of pid = 1 is updated to 6, so if the SQL statement is executed repeatedly, it will not take effect because the data of pid = 1 AND version = 5 is no longer valid. exists, so that the idempotence of updates can be maintained, and multiple updates will not affect the results.

Anti-duplication Token Token

Program description:

For situations such as continuous clicks by the client or timeout retries by the caller, such as order submission, the Token mechanism can be used to prevent repeated submissions. To put it simply, the caller first requests a global ID (Token) from the backend when calling the interface. When requesting, it carries this global ID with it (it is best to put the Token in Headers). The backend needs to use this Token. As Key, the user information is sent to Redis as Value for key value content verification. If the Key exists and the Value matches, the delete command is executed, and then the subsequent business logic is executed normally. If there is no corresponding Key or Value does not match, a repeated error message will be returned to ensure idempotent operations.

Applicable operations:

• insert operation;
• update operation;
• delete operation.

Usage restrictions:

• A globally unique Token string needs to be generated;
• The third-party component Redis needs to be used for data validation.

Main process:

• The server provides an interface for obtaining Token. The Token can be a serial number, a distributed ID or a UUID string;

• The client calls the interface to obtain the Token. At this time, the server will generate a Token string;

• Then store the string in the Redis database, using the Token as the Redis key (note that the expiration time is set);

• Return the Token to the client. After the client obtains it, it should be stored in the hidden field of the form;

• When the client executes and submits the form, it stores the Token into the Headers and carries the Headers with it when executing the business request;

• After receiving the request, the server gets the Token from the Headers, and then checks whether the key exists in Redis based on the Token;

• The server determines whether the key exists in Redis. If it exists, it deletes the key and then executes the business logic normally. If it does not exist, an exception will be thrown and an error message for repeated submissions will be returned.

Note that under concurrent conditions, atomicity needs to be ensured when executing Redis data search and deletion, otherwise idempotence may not be guaranteed under concurrency. The implementation method can use distributed locks or use Lua expressions to log out query and delete operations.

Pass unique sequence number downstream

Program description:

The so-called request sequence number actually means that each request to the server is accompanied by a unique and non-repeating sequence number in a short period of time. The sequence number can be a sequential ID or an order number. It is generally generated by the downstream. When calling The upstream server interface appends the serial number and the ID used for authentication. When the upstream server receives the request information, it combines the serial number and the downstream authentication ID to form a Key for operating Redis, and then queries Redis to see whether there is a key-value pair for the corresponding Key. According to the result:

• If it exists, it means that the downstream request for the sequence number has been processed. At this time, you can directly respond to the error message of the repeated request;
• If it does not exist, use the Key as the key of Redis, use the downstream key information as the stored value (such as some business logic information passed by the downstream provider), store the key-value pair in Redis, and then execute the corresponding business normally. Just logic.

Applicable operations:

• insert operation;
• update operation;
• delete operation.

Usage restrictions:

• Requiring third parties to pass unique serial numbers;
• The third-party component Redis needs to be used for data validation.

Main process:

The main steps:

• The downstream service generates the distributed ID as a serial number, and then executes the request to call the upstream interface, along with the "unique serial number" and the requested "authentication credential ID";
• The upstream service performs security verification and detects whether there is a "serial number" and "credential ID" in the parameters passed downstream;
• The upstream service goes to Redis to detect whether there is a Key composed of the corresponding "serial number" and "authentication ID". If it exists, it will throw a repeated execution exception message, and then respond to the corresponding error message from the downstream. If it does not exist, use the combination of "serial number" and "authentication ID" as the key, use the downstream key information as the value, and then store it in Redis, and then execute the subsequent business logic normally.

In the above steps, when inserting data into Redis, you must set the expiration time. This ensures that within this time range, if the interface is called repeatedly, judgment and identification can be made. If the expiration time is not set, it is likely that an unlimited amount of data will be stored in Redis, causing Redis to not work properly.

7. Example of idempotent interface

The anti-duplication token scheme is used here, which can ensure idempotence under different request actions. For the implementation logic, you can see the "anti-duplication token" scheme written above. Next, write the code to implement this logic.

Maven introduces related dependencies

The Maven tool is used here to manage dependencies, and SpringBoot, Redis, and lombok related dependencies are introduced in pom.xml.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.4.RELEASE</version>
    </parent>

    <groupId>mydlq.club</groupId>
    <artifactId>springboot-idempotent-token</artifactId>
    <version>0.0.1</version>
    <name>springboot-idempotent-token</name>
    <description>Idempotent Demo</description>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <!--springboot web-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <!--springboot data redis-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-pool2</artifactId>
        </dependency>
        <!--lombok-->
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Configure parameters for connecting to Redis

Configure the parameters for connecting to Redis in the application configuration file. I won’t introduce the basics of Spring Boot. For the latest tutorials, I recommend reading the tutorial below.

as follows:

spring:
  redis:
    ssl: false
    host: 127.0.0.1
    port: 6379
    database: 0
    timeout: 1000
    password:
    lettuce:
      pool:
        max-active: 100
        max-wait: -1
        min-idle: 0
        max-idle: 20

Create and verify Token tool class

Create a Service class for operating Token, which contains Token creation and verification methods, among which:

• Token creation method: Use the UUID tool to create a Token string, set "idempotent_token:" + "Token string" as the Key, use user information as the Value, and store the information in Redis.
• Token verification method: receive the Token string parameter, add the Key prefix to form the Key, then pass in the value value, and execute the Lua expression (Lua expression can ensure the atomicity of command execution) to find the corresponding Key and delete operations. After the execution is completed, verify the return result of the command. If the result is not empty and non-0, the verification is successful, otherwise it fails.

import java.util.Arrays;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.stereotype.Service;

@Slf4j
@Service
public class TokenUtilService {
    
    

    @Autowired
    private StringRedisTemplate redisTemplate;

    /**
     * 存入 Redis 的 Token 键的前缀
     */
    private static final String IDEMPOTENT_TOKEN_PREFIX = "idempotent_token:";

    /**
     * 创建 Token 存入 Redis，并返回该 Token
     *
     * @param value 用于辅助验证的 value 值
     * @return 生成的 Token 串
     */
    public String generateToken(String value) {
    
    
        // 实例化生成 ID 工具对象
        String token = UUID.randomUUID().toString();
        // 设置存入 Redis 的 Key
        String key = IDEMPOTENT_TOKEN_PREFIX + token;
        // 存储 Token 到 Redis，且设置过期时间为5分钟
        redisTemplate.opsForValue().set(key, value, 5, TimeUnit.MINUTES);
        // 返回 Token
        return token;
    }

    /**
     * 验证 Token 正确性
     *
     * @param token token 字符串
     * @param value value 存储在Redis中的辅助验证信息
     * @return 验证结果
     */
    public boolean validToken(String token, String value) {
    
    
        // 设置 Lua 脚本，其中 KEYS[1] 是 key，KEYS[2] 是 value
        String script = "if redis.call('get', KEYS[1]) == KEYS[2] then return redis.call('del', KEYS[1]) else return 0 end";
        RedisScript<Long> redisScript = new DefaultRedisScript<>(script, Long.class);
        // 根据 Key 前缀拼接 Key
        String key = IDEMPOTENT_TOKEN_PREFIX + token;
        // 执行 Lua 脚本
        Long result = redisTemplate.execute(redisScript, Arrays.asList(key, value));
        // 根据返回结果判断是否成功成功匹配并删除 Redis 键值对，若果结果不为空和0，则验证通过
        if (result != null && result != 0L) {
    
    
            log.info("验证 token={},key={},value={} 成功", token, key, value);
            return true;
        }
        log.info("验证 token={},key={},value={} 失败", token, key, value);
        return false;
    }

}

Create a test Controller class

Create a Controller class for testing, which has an interface for obtaining Token and testing the idempotence of the interface. The content is as follows:

import lombok.extern.slf4j.Slf4j;
import mydlq.club.example.service.TokenUtilService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

@Slf4j
@RestController
public class TokenController {
    
    

    @Autowired
    private TokenUtilService tokenService;

    /**
     * 获取 Token 接口
     *
     * @return Token 串
     */
    @GetMapping("/token")
    public String getToken() {
    
    
        // 获取用户信息（这里使用模拟数据）
        // 注：这里存储该内容只是举例，其作用为辅助验证，使其验证逻辑更安全，如这里存储用户信息，其目的为:
        // - 1)、使用"token"验证 Redis 中是否存在对应的 Key
        // - 2)、使用"用户信息"验证 Redis 的 Value 是否匹配。
        String userInfo = "mydlq";
        // 获取 Token 字符串，并返回
        return tokenService.generateToken(userInfo);
    }

    /**
     * 接口幂等性测试接口
     *
     * @param token 幂等 Token 串
     * @return 执行结果
     */
    @PostMapping("/test")
    public String test(@RequestHeader(value = "token") String token) {
    
    
        // 获取用户信息（这里使用模拟数据）
        String userInfo = "mydlq";
        // 根据 Token 和与用户相关的信息到 Redis 验证是否存在对应的信息
        boolean result = tokenService.validToken(token, userInfo);
        // 根据验证结果响应不同信息
        return result ? "正常调用" : "重复调用";
    }

}

Create SpringBoot startup class

Create a startup class to start the SpringBoot application. I won’t introduce the basic tutorial. It is recommended to read the following tutorial, which is very complete.

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Application {
    
    

    public static void main(String[] args) {
    
    
        SpringApplication.run(Application.class, args);
    }

}

Write test classes for testing

Write a test class to test, access the same interface multiple times, and test whether it can be executed successfully only the first time.

import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.http.MediaType;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.request.MockMvcRequestBuilders;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import org.springframework.web.context.WebApplicationContext;

@Slf4j
@SpringBootTest
@RunWith(SpringRunner.class)
public class IdempotenceTest {
    
    

    @Autowired
    private WebApplicationContext webApplicationContext;

    @Test
    public void interfaceIdempotenceTest() throws Exception {
    
    
        // 初始化 MockMvc
        MockMvc mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext).build();
        // 调用获取 Token 接口
        String token = mockMvc.perform(MockMvcRequestBuilders.get("/token")
                .accept(MediaType.TEXT_HTML))
                .andReturn()
                .getResponse().getContentAsString();
        log.info("获取的 Token 串：{}", token);
        // 循环调用 5 次进行测试
        for (int i = 1; i <= 5; i++) {
    
    
            log.info("第{}次调用测试接口", i);
            // 调用验证接口并打印结果
            String result = mockMvc.perform(MockMvcRequestBuilders.post("/test")
                    .header("token", token)
                    .accept(MediaType.TEXT_HTML))
                    .andReturn().getResponse().getContentAsString();
            log.info(result);
            // 结果断言
            if (i == 0) {
    
    
                Assert.assertEquals(result, "正常调用");
            } else {
    
    
                Assert.assertEquals(result, "重复调用");
            }
        }
    }

}

Displayed as follows:

[main] IdempotenceTest:  获取的 Token 串：980ea707-ce2e-456e-a059-0a03332110b4
[main] IdempotenceTest:  第1次调用测试接口
[main] IdempotenceTest:  正常调用
[main] IdempotenceTest:  第2次调用测试接口
[main] IdempotenceTest:  重复调用
[main] IdempotenceTest:  第3次调用测试接口
[main] IdempotenceTest:  重复调用
[main] IdempotenceTest:  第4次调用测试接口
[main] IdempotenceTest:  重复调用
[main] IdempotenceTest:  第5次调用测试接口
[main] IdempotenceTest:  重复调用

8. Summary

Idempotence is a very common and important requirement in development, especially for services linked to money such as payments and orders. It is especially important to ensure the idempotence of interfaces. In actual development, we need to flexibly choose idempotent implementation methods for different business scenarios:

• For orders that have a unique primary key, the "unique primary key solution" can be used;
• For related update scenario operations such as updating order status, it is easier to implement using the "optimistic locking scheme";
• For upstream and downstream situations, where the downstream requests the upstream, the upstream service can use the "downstream delivery unique sequence number scheme" which is more reasonable;
• Similar to the scenario where the front-end repeatedly submits, places orders repeatedly, and does not have a unique ID number, it can be implemented more quickly through the "anti-duplication Token solution" that cooperates with Token and Redis;

The above are just some suggestions. I would like to emphasize again that to achieve idempotence, you need to first understand your own business needs and implement it according to business logic. Only by properly handling the details of each node and improving the overall business process design can you achieve better results. Good to ensure the normal operation of the system.

Scheme name	Applicable method	Implementation complexity	Disadvantages of the solution
Database unique primary key	insert operation delete operation	Simple	- Can only be used for insert operations; - Can only be used in scenarios where there is a unique primary key;
Database optimistic locking	update operation	Simple	- Can only be used for update operations; - Additional fields need to be added to the table;
Request serial number	insert operation update operation delete operation	Simple	- Need to ensure that the downstream generates a unique serial number; - Redis is required as a third party to store the requested serial number;
Anti-duplication Token Token	insert operation update operation delete operation	Moderate	- Token string generated by Redis third-party storage is required;