1. What is idempotence?
In mathematics, idempotence means that the results of multiple operations are consistent. Corresponding to the actual working software or network environment, the result of the same operation is the same no matter how many times you operate it.
In the programming process, we will see that some idempotence naturally exists, such as:
- select query operation
- In the delete operation, delete based on a certain key value.
- update updates a field value
2. Why does the idempotence problem occur?
The reason why idempotent problems occur is nothing more than repeated clicks or network resends, for example:
1) Click the submit button twice
2) Click the refresh button while the operation is in progress
3) Repeat the previous operation after going back in the browser, resulting in duplication Submit the form
4) Nginx resend
5) Try resend in distributed RPC environment
6) Repeated consumption of messages. When using MQ message middleware, the message middleware error is not submitted in time, resulting in repeated consumption.
3. Ensure idempotent solutions
In order to ensure idempotence, there are mainly the following methods:
1) Anti-duplication identifier (Token token) implementation
This method is that the caller first requests a global ID (Token) from the backend when calling the interface, and carries the global ID with the request. The backend needs to use this Token as the Key, and the user information as the Value to Redis for key-value content. Verify, if the Key exists and the Value matches, execute the delete command, and then execute the subsequent business logic. If the corresponding Key does not exist or the Value does not match, an execution error message will be returned.
The usage process is shown in the figure below:
①The server provides an interface for obtaining Token. This Token can be a serial number, distributed ID or UUID. The client calls the interface to obtain the Token, and the server will generate a Token string.
②Save this Token string into Redis, and use the Token as the Redis key (the expiration time needs to be set).
③Return the Token to the client, and the client stores it in the hidden field of the form after getting it.
④When the client executes and submits the form, it brings the Token in the Header.
⑤The server receives the request, gets the Token from the Header, and then searches Redis to see if the corresponding Key exists. If it exists, delete the Key. If it does not exist, a duplicate submission exception will be thrown. It should be noted here that the search and delete operations must ensure atomicity, otherwise idempotence may not be guaranteed in concurrent situations. As for atomicity, query and delete operations can be logged out through distributed locks or Lua scripts.
⑥ Return the result, execute normal business logic or prompt an error message.
This method can be applied to insert, update and delete operations. The limitation is that a globally unique Token string needs to be generated, and Redis needs to be used for data verification.
Here we take a detailed look at its implementation method:
pom implementation
introduces springboot, Redis, lombok and other related dependencies
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>2.3.5.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
<version>2.2.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.6.1</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.16</version>
</dependency>
</dependencies>
application implements
a Redis connection related parameter configuration file
spring:
redis:
ssl: false
host: 127.0.0.1
port: 6379
database: 0
timeout: 1000
password:
lettuce:
pool:
max-active: 100
max-wait: -1
min-idle: 0
max-idle: 20
Create Token verification Token tool class
@Slf4j
@Service
public class TokenUtilService {
@Autowired
private StringRedisTemplate redisTemplate;
/*
* 存入Redis的Token的前缀
*/
private static final String IDEMPOTENT_TOKEN_PREFIX = "idempotent_token:";
public String generateToken(String value) {
String token = UUID.randomUUID().toString();
//设置存入Redis的key
String key = IDEMPOTENT_TOKEN_PREFIX + token;
//存储Token到Redis并设置过期时间为5分钟
redisTemplate.opsForValue().set(key, value, 5, TimeUnit.MiNUTES);
return token;
}
public boolean validToken(String token, String value) {
//设置Lus脚本,KEYS[1]是key,KEYS[2]是value
String script = "if redis.call('get', KEYS[1]) == KEYS[2] then return redis.call('del', KEYS[1]) else return 0 end";
RedisScript<Long> redisScript= new DefaultRedisScript<>(script, Long.class);
String key = IDEMPOTENT_TOKEN_PREFIX + token;
//执行Lua脚本
Long result = redisTemplate.execute(redisScript, Arrays.asList(key, value));
//根据返回结果判断是否成功匹配并删除，结果不为空或0，验证通过
if (result != null && result != 0L) {
log.info("验证 token={}, key={}, value={}, 成功", token, key, value);
return true;
}
log.info("验证 token={}, key={}, value={}, 失败", token, key, value);
return false;
}
}
Test class (Controller layer simulation)
@Slf4j
@RestController
public class TokenContoller {
@Autowired
private TokenUtilService tokenService;
/*
* 获取Token接口，返回Token串
*/
@GetMapping("/token")
public String getToken() {
//模拟数据，使用token验证是否存在对应的key
String userInfo = "myInfo";
return tokenService.generateToken(userInfo);
}
/*
* 幂等性测试接口
*/
@PostMapping("/test")
public String test(@RequestHeader(value = "token") String token) {
String userInfo = "myInfo";
boolean result = tokenService.validToken(tolen, userInfo);
return result ? "正常调用":"重复调用";
}
}
Finally, there is an improved version of this solution, which is to introduce the relational library and use the transaction characteristics of the relational library to ensure the atomicity of the operation. That is to insert the processed data into the relational library, and finally insert the idempotent Key into Redis. , which can still guarantee idempotence under concurrent conditions.
2) Implementation of unique serial number transmission downstream
Each request to the server is accompanied by a short-term unique and non-repeating sequence number. This sequence number is generally generated downstream. When calling the upstream server interface, the sequence number and ID used for authentication are appended. The upstream server combines this serial number with the downstream authentication ID to form a key used to operate Redis, and then queries Redis to see whether the corresponding key exists. If it exists, it means that the downstream sequence number request has been processed and the error message of the repeated request is directly returned; if it does not exist, this Key is used as the Redis key, and the downstream key information is used as the stored value, and the key is The value pairs are stored in Redis and then normal business logic is executed.
The process used is shown below:
It should be noted that when inserting data into Redis, the expiration time must be set. This ensures that repeated calls to the interface can be identified within the time range, otherwise an unlimited amount of data may be stored in Redis.
This method is suitable for insert, update and delete operations, at the cost of requiring a third party to pass a unique sequence number, and using Redis for data verification.
3) Implemented with the help of database primary key
The constraint feature of the database's unique primary key is used here. This method is suitable for idempotence during insertion and can ensure that a table value stores a record with the primary key. The primary key used here generally refers to the distributed ID, so that Ensure the global uniqueness of ID in a distributed environment.
The usage process is shown in the figure below:
①The client executes the creation request and calls the server interface.
② The server executes the business logic and generates a distributed ID, using the ID as the primary key of the inserted data to perform the insertion operation. The ID generation algorithm here can use the snowflake algorithm, or use the database number segment mode or the Redis auto-increment method to generate distribution formula unique ID.
③The server performs the insertion into the database. If the insertion is successful, it means that the interface is not called repeatedly. If a primary key duplication exception is thrown, an error message is returned to the client.
This method is suitable for insertion and deletion operations, with the limitation that a primary key needs to be generated.
4) Use database optimistic locking
Database optimistic locking is generally used for update operations by adding a version identification field to the corresponding database table, so that the version identification value will be checked for each update.
Its usage process is very simple as shown below:
The only thing that needs to be paid attention to is that there is one more condition to judge the current version when executing the update statement, for example:
update my_table set price=price+50, version=version+1 where id = 3 and version = 5 ;
In this way, the version will change every time it is executed. If it is executed repeatedly, the original version number will not take effect, ensuring idempotence.
This method can only be used for update operations, and it also requires adding an additional field to the corresponding database table.
Finally, we summarize the four commonly used backend methods for dealing with idempotence issues, as follows:
In addition to the main methods mentioned above, there are other methods that can also be used:
5) With the help of local lock
The ConcurrentHashMap concurrent container putIfAbsent method and the ScheduledThreadPoolExecutor timing task are used. You can also use the guava cache mechanism. It is also possible to have a cached effective time in guava. The key is generated through Content-MD5. Content-MD5 is unique within a certain range. , it can be considered approximately unique when used, and can be used as a key in a low-concurrency environment.
Of course, local locks are only applicable to applications deployed on a single machine. Let’s take a look at its simple implementation:
Configuration annotations:
import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface Resubmit {
/*
* 延时时间，在延时多久后可以再次提交，单位为秒
* */
int delaySeconds() default 20;
}
Instantiate the lock:
import com.google.common.cache.Cache;
import com.google.common.cache.CacheBuilder;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.codec.digest.DigestUtils;
import java.util.Objects;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
@Slf4j
public final class ResubmitLock {
private static final ConcurrentHashMap LOCK_CACHE = new ConcurrentHashMap(200);
private static final ScheduledThreadPoolExecutor EXECUTOR = new ScheduledThreadPoolExecutor(5, new ThreadPoolExecutor.DiscardPolicy());
private ResubmitLock() {
}
/*
* 静态内部类的单例模式
* */
private static class SingletonInstance {
private static final ResubmitLock Instance = new ResubmitLock();
}
public static ResubmitLock getInstance() {
return SingletonInstance.Instance;
}
public static String handleKey(String param) {
return DigestUtils.md5Hex(param == null ? "" : param);
}
public boolean lock(final String key, Object value) {
return Objects.isNull(LOCK_CACHE.putIfAbsent(key, value));
}
public void unlock(final boolean lock, final String key, final int delaySeconds) {
if (lock) {
EXECUTOR.schedule(() -> {
LOCK_CACHE.remove(key);
}, delaySeconds, TimeUnit.SECONDS);
}
}
}
AOP aspects:
import java.lang.reflect.Method;
@Log4j
@Aspect
@Component
public class ResubmitDataAspect {
private final static String DATA = "data";
private final static Object PRESENT = new Object();
@Around("@annotation(com.cn.xxx.common.annotation.Resubmit)")
public Object handleResubmit(ProceedingJoinPoint joinPoint) {
Method method = ((MethodSignature) joinPoint.getSignature()).getMethod();
//获取注解信息
Resubmit annotation = method.getAnnotation(Resubmit.class);
int delaySeconds = annotation.delaySeconds();
Object[] pointArgs = joinPoint.getArgs();
String key = "";
//获取第一个参数
Object firstParam = pointArgs[0];
if (firstParam instanceof RequestDTO) {
//解析参数
JSONObject requestDTO = JSONObject.parseObject(firstParam.toString());
JSONObject data = JSONObject.parseObject(requestDTO.getString(DATA));
if (data != null) {
StringBuffer sb = new StringBuffer();
data.forEach((k, v) -> {
sb.apperd(v);
});
key = ResubmitLock.handleKey(sb.toString());
}
}
boolean lock = false;
try {
//设置解锁key
lock = ResubmitLock.getInstance().lock(key, PRESENT);
if (lock) {
//放行
return joinPoint.proceed();
} else {
//响应重复提交异常
return new ResponseDTO<>(RespoinseCode.REPEAT_SUBMIT_OPERATION_EXCEPTION);
}
} finally {
//设置解锁key和解锁时间
ResubmitLock.getInstance().unLock(lock, key, delaySeconds);
}
}
}
Usage notes:
public class ResponseToSavaPosts {
@ApiOperation(value = "保存我的帖子接口", notes = "保存我的帖子接口")
@PostMapping("/posts/save")
@Resubmit(delaySeconds = 10)
public void ResponseToSava(@RequestBody @Validated RequestDTOrequestDto) {
return bbsPostsBizService.saveBbsPosts(requestDto);
}
}
6) With the help of distributed Redis lock
Anyone familiar with Redis knows that it is thread-safe. We can easily implement a distributed lock by using its features, such as opsForValue().setIfAbsent(key). Its function is to cache and return at the same time if there is no current key in the cache. true, after caching, set an expiration time for the key to prevent the lock from being released due to system crashes and causing a deadlock. We can think that when true is returned, he has obtained the lock. When the lock is not released, we perform an exception. Throw.
7) Use database pessimistic locking
Use select ... for update, which has the same principle as synchronized. It locks first and then checks and then performs the update or insert operation. The problem with this is to consider how to avoid deadlock, and the efficiency is relatively poor. This method can be used when the concurrency of a single application is small.
8) Front-end page guarantee
Usually after submission, the submit button is set to prohibit clicking (usually a fixed time period is set).
9) Use Post/Redirect/Get mode
This method is to perform page redirection after submission, PRG (Post-Redirect-Get) mode.
That is to say, after the user submits the form, a client-side redirection is performed to the information page of successful submission. This can avoid repeated submissions caused by page refreshes, and there will be no warnings about repeated submissions of browser forms, and it can also eliminate problems caused by pressing the browser's forward and back buttons.