[Technical Application] Java interface idempotence implementation plan

I. Introduction

I am working on an online project recently, and there are still many differences from previous intranet projects, especially in terms of security and concurrency processing. More measures need to be taken. The first step is to 接口的幂等性go up. This is also the interface for concurrent request security. To protect the bottom line, based on actual business scenarios, we summarized and implemented a demo of interface idempotence, which I would like to share here;

2. Idempotence

1. Concept:

• Idempotence is originally a mathematical concept, that is, a formula: f(x)=f(f(x))a mathematical property that can be established. When used in the field of programming, it means that for the same system, using the same conditions, one request and repeated multiple requests have the same impact on system resources .
• Idempotence is a very important concept in distributed system design. Interfaces with this property are always designed with the concept that when an exception occurs when calling the interface and repeated attempts are made, it will always cause unbearable consequences for the system. losses, so this phenomenon must be prevented from happening .
• There are many ways to achieve idempotence, and currently the request token mechanism has a wide range of applications. The core idea is to generate a unique credential, that is, a token, for each operation. A token has only one execution right at each stage of the operation, and once the execution is successful, the execution result is saved. For repeated requests, the same result (error report) is returned, etc.

2. Reasons for interface idempotence design

1) The front-end repeatedly submits the form.
When filling in some forms, the user completes the submission. In many cases, due to network fluctuations, the user does not respond to the successful submission in time, causing the user to think that the submission was not successful, and then keeps clicking the submit button. At this time, A duplicate form request has occurred.

2) Malicious attack by hackers.
For example, when implementing the user voting function, if a hacker repeatedly submits votes for a user, this will cause the interface to receive voting information repeatedly submitted by the user, which will cause the voting results to be seriously inconsistent with the facts.

3) Interface timeout and repeated submission.
Most RPC frameworks [such as Dubbo], in order to prevent request failures caused by network fluctuations and timeouts, will add a retry mechanism, resulting in a request being submitted multiple times.

4) Repeated consumption of messages
When using MQ message middleware, if the Consumer consumption times out or the producer sends a message but does not receive an ACK due to network reasons and the message is resent, it will lead to repeated consumption.

3. Which interfaces need to be idempotent?

The implementation and judgment of idempotence requires certain resources. Therefore, idempotence judgment should not be added to each interface. It should be distinguished according to the actual business situation and operation type. For example, we do not need to make idempotence judgments when performing query operations and deletion operations.

The results of the query operation are the same whether checked once or multiple times, so we do not need to make idempotent judgments. The same is true for deletion operations. Deletion once and deletion multiple times will delete the relevant data (deletion here refers to conditional deletion rather than deletion of all data), so there is no need to make idempotent judgments.

So which interfaces need idempotence? Regarding this issue, we need to start from the specific business, but there are also rules that can be followed as follows:

3. Idempotent design ideas

1. Before the request starts, the result is found according to the key query: error is reported and no result is found: save key-value-expireTime
   key=ip+url+args
2. After the request is completed, 删除 keyit is directly deleted regardless of whether the key exists or not. It is configurable.
3. expireTimeThe expiration time prevents a request from being stuck. 阻塞If it exceeds the expiration time, the expiration time will be automatically deleted. The expiration time is greater than the business execution time, so it needs to be roughly evaluated;
4. This solution directly cuts to the interface request level.
5. The expiration time needs to be greater than the business execution time. Otherwise, business request 1 is still being executed and the front end is not masked, or the user jumps to the page and then comes back to make repeated request
   2. From a business perspective, the result is still not in line with expectations. .
6. Suggestions delKey = false. Even if the business is executed, the key will not be deleted and the lock expireTimetime will be forced. Prevent 5 from happening.
7. Implementation idea : For requests with the same request IP and interface, and the same parameters, multiple requests within expireTime are only allowed to succeed once.
8. Page masking, unique indexing at the database level, querying first and then adding, and other processing methods should be dealt with.
9. This design is only used for idempotence and not for locks. 100 concurrent stress tests will cause problems and are meaningless in this scenario. In practice, users will not manually send 50 or 50 messages in 1s or 3s. 100 repeated requests, or 100 repeated requests under a weak network;

4. Implementation code

1. Custom annotations

Custom annotationsIdempotent

package com.sk.idempotenttest.annotation;

import java.lang.annotation.*;
import java.util.concurrent.TimeUnit;

/**
 * @description: Idempotent annotation
 *
 * @author dylan
 *
 */
@Inherited
@Target(ElementType.METHOD)
@Retention(value = RetentionPolicy.RUNTIME)
public @interface Idempotent {
    
    


    /**
     * 是否做幂等处理
     * false：非幂等
     * true：幂等
     * @return
     */
    boolean isIdempotent() default false;

    /**
     * 有效期
     * 默认：1
     * 有效期要大于程序执行时间，否则请求还是可能会进来
     * @return
     */
    int expireTime() default 1;

    /**
     * 时间单位
     * 默认：s
     * @return
     */
    TimeUnit timeUnit() default TimeUnit.SECONDS;

    /**
     * 提示信息，可自定义
     * @return
     */
    String info() default "重复请求，请稍后重试";

    /**
     * 是否在业务完成后删除key
     * true:删除
     * false:不删除
     * @return
     */
    boolean delKey() default false;
}

Annotation usage:

@Idempotent(isIdempotent = true,expireTime = 10,timeUnit = TimeUnit.SECONDS,info = "请勿重复请求",delKey = false)
@GetMapping("/test")
public String test(){
    
    
  return "哈哈哈";
}

Annotation attribute description:

• key: The unique identifier of the idempotent operation, use spring el expression to reference method parameters with #. If it is empty, url + args
  the unique identifier of the current request is taken.
• expireTime: Validity period 默认：1The validity period must be greater than the program execution time, otherwise the request may still come in
• timeUnit: Default time unit:s （秒）
• info: idempotent failure prompt message, customizable
• delKey: Whether to delete the key after the business is completed true: delete false: not delete

2. Aspect realization

package com.sk.idempotenttest.aspect;

import com.sk.idempotenttest.annotation.Idempotent;
import com.sk.idempotenttest.exception.IdempotentException;
import com.sk.idempotenttest.utils.ServerTool;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.aspectj.lang.JoinPoint;
import org.aspectj.lang.annotation.After;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Before;
import org.aspectj.lang.annotation.Pointcut;
import org.aspectj.lang.reflect.MethodSignature;
import org.redisson.Redisson;
import org.redisson.api.RMapCache;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.util.CollectionUtils;
import org.springframework.web.context.request.RequestContextHolder;
import org.springframework.web.context.request.ServletRequestAttributes;
import javax.servlet.http.HttpServletRequest;
import java.lang.reflect.Method;
import java.time.LocalDateTime;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

/**
 * @description: The Idempotent Aspect
 *
 * @author dylan
 *
 */
@Slf4j
@Aspect
@Component
@RequiredArgsConstructor
public class IdempotentAspect {
    
    

    private ThreadLocal<Map<String,Object>> threadLocal = new ThreadLocal();
    private static final String RMAPCACHE_KEY = "idempotent";
    private static final String KEY = "key";
    private static final String DELKEY = "delKey";

    private final Redisson redisson;


    @Pointcut("@annotation(com.sk.idempotenttest.annotation.Idempotent)")
    public void pointCut(){
    
    }

    @Before("pointCut()")
    public void beforePointCut(JoinPoint joinPoint)throws Exception{
    
    
        ServletRequestAttributes requestAttributes =
                (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
        HttpServletRequest request = requestAttributes.getRequest();

        MethodSignature signature = (MethodSignature)joinPoint.getSignature();
        Method method = signature.getMethod();
        if(!method.isAnnotationPresent(Idempotent.class)){
    
    
            return;
        }
        Idempotent idempotent = method.getAnnotation(Idempotent.class);
        boolean isIdempotent = idempotent.isIdempotent();
        if(!isIdempotent){
    
    
            return;
        }

        String ip = ServerTool.getIpAddress(request);
        String url = request.getRequestURL().toString();
        String argString  = Arrays.asList(joinPoint.getArgs()).toString();
        String key = ip+ url + argString;  //key主要用于对请求者，请求客户端的请求频次做限制，也可以使用token作为key

        long expireTime = idempotent.expireTime();
        String info = idempotent.info();
        TimeUnit timeUnit = idempotent.timeUnit();
        boolean delKey = idempotent.delKey();

        //do not need check null
        RMapCache<String, Object> rMapCache = redisson.getMapCache(RMAPCACHE_KEY);
        String value = LocalDateTime.now().toString().replace("T", " ");
        Object v1;
        if (null != rMapCache.get(key)){
    
    
            //had stored
            throw new IdempotentException(info);
        }
        synchronized (this){
    
    
            v1 = rMapCache.putIfAbsent(key, value, expireTime, TimeUnit.SECONDS);
            if(null != v1){
    
    
                throw new IdempotentException(info);
            }else {
    
    
                log.info("[idempotent]:has stored key={},value={},expireTime={}{},now={}",key,value,expireTime,timeUnit,LocalDateTime.now().toString());
            }
        }

        Map<String, Object> map =
                CollectionUtils.isEmpty(threadLocal.get()) ? new HashMap<>(4):threadLocal.get();
        map.put(KEY,key);
        map.put(DELKEY,delKey);
        threadLocal.set(map);

    }

    @After("pointCut()")
    public void afterPointCut(JoinPoint joinPoint){
    
    
        Map<String,Object> map = threadLocal.get();
        if(CollectionUtils.isEmpty(map)){
    
    
            return;
        }

        RMapCache<Object, Object> mapCache = redisson.getMapCache(RMAPCACHE_KEY);
        if(mapCache.size() == 0){
    
    
            return;
        }

        String key = map.get(KEY).toString();
        boolean delKey = (boolean)map.get(DELKEY);

        if(delKey){
    
    
            mapCache.fastRemove(key);
            log.info("[idempotent]:has removed key={}",key);
        }
        threadLocal.remove();
    }
}

3. Global exception handling

GlobalExceptionHandler kind

package com.sk.idempotenttest.exception;

import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.ResponseStatus;

@Slf4j
@ControllerAdvice
@ResponseBody
public class GlobalExceptionHandler {
    
    

    @ExceptionHandler(IdempotentException.class)
    @ResponseStatus(value= HttpStatus.BAD_REQUEST)
    public JsonResult handleHttpMessageNotReadableException(IdempotentException ex){
    
    
        log.error("请求异常,{}",ex.getMessage());
        return new JsonResult("400",ex.getMessage());
    }

}

Return value JsonResultclass

package com.sk.idempotenttest.exception;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@NoArgsConstructor
@AllArgsConstructor
public class JsonResult {
    
    

    private String code;//状态码
    private String msg;//请求信息

}

4. redis stores request information

RedissonConfigkind

package com.sk.idempotenttest.config;

import org.redisson.Redisson;
import org.redisson.config.Config;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * @description: Redisson配置类
 *
 * @author ITyunqing
 * @since 1.0.0
 */
@Configuration
public class RedissonConfig {
    
    

    @Value("${singleServerConfig.address}")
    private String address;

    @Value("${singleServerConfig.password}")
    private String password;

    @Value("${singleServerConfig.pingTimeout}")
    private int pingTimeout;

    @Value("${singleServerConfig.connectTimeout}")
    private int connectTimeout;

    @Value("${singleServerConfig.timeout}")
    private int timeout;

    @Value("${singleServerConfig.idleConnectionTimeout}")
    private int idleConnectionTimeout;

    @Value("${singleServerConfig.retryAttempts}")
    private int retryAttempts;

    @Value("${singleServerConfig.retryInterval}")
    private int retryInterval;

    @Value("${singleServerConfig.reconnectionTimeout}")
    private int reconnectionTimeout;

    @Value("${singleServerConfig.failedAttempts}")
    private int failedAttempts;

    @Value("${singleServerConfig.subscriptionsPerConnection}")
    private int subscriptionsPerConnection;

    @Value("${singleServerConfig.subscriptionConnectionMinimumIdleSize}")
    private int subscriptionConnectionMinimumIdleSize;

    @Value("${singleServerConfig.subscriptionConnectionPoolSize}")
    private int subscriptionConnectionPoolSize;

    @Value("${singleServerConfig.connectionMinimumIdleSize}")
    private int connectionMinimumIdleSize;

    @Value("${singleServerConfig.connectionPoolSize}")
    private int connectionPoolSize;


    @Bean(destroyMethod = "shutdown")
    public Redisson redisson() {
    
    
        Config config = new Config();
        config.useSingleServer().setAddress(address)
                .setPassword(password)
                .setIdleConnectionTimeout(idleConnectionTimeout)
                .setConnectTimeout(connectTimeout)
                .setTimeout(timeout)
                .setRetryAttempts(retryAttempts)
                .setRetryInterval(retryInterval)
                .setReconnectionTimeout(reconnectionTimeout)
                .setPingTimeout(pingTimeout)
                .setFailedAttempts(failedAttempts)
                .setSubscriptionsPerConnection(subscriptionsPerConnection)
                .setSubscriptionConnectionMinimumIdleSize(subscriptionConnectionMinimumIdleSize)
                .setSubscriptionConnectionPoolSize(subscriptionConnectionPoolSize)
                .setConnectionMinimumIdleSize(connectionMinimumIdleSize)
                .setConnectionPoolSize(connectionPoolSize);
        return (Redisson) Redisson.create(config);
    }

}

5、pom.xml

<!--aop-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
        <!--redisson-->
        <dependency>
            <groupId>org.redisson</groupId>
            <artifactId>redisson</artifactId>
            <version>3.5.4</version>
        </dependency>

V. Summary

幂等性Not only can it ensure the normal execution of the program, but it can also prevent some junk data and invalid requests from consuming system resources. It is recommended 分布式锁to use , as this solution is more versatile. A summary of distributed locks will be introduced later .

suggestion:

• Lua 脚本It is recommended to implement the code logic of whether token exists in redis and delete it to ensure atomicity.
• A globally unique ID can be generated 百度的 uid-generatorusing美团的 Leaf

==If the article is helpful to you, please like, collect and leave a review=