Use unique request number to remove duplicates
What you might think is that as long as the request has a unique request number, you can borrow Redis to do this deduplication-as long as the unique request number exists in redis and proves that it has been processed, then it is considered a duplicate
The code is roughly as follows:
String KEY = "REQ12343456788";//Request a unique number
long expireTime = 1000; // Expires in 1000 milliseconds, repeated requests within 1000 milliseconds will be considered repeated
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
//If the redis key still exists, consider the request as a duplicate
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection -> connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
final boolean isConsiderDup;
if (firstSet != null && firstSet) {// first visit
isConsiderDup = false;
} else {// The redis value already exists and is considered a duplicate
isConsiderDup = true;
}
Deduplication of business parameters
The above solution can solve the scenario with a unique request number. For example, before each write request, the server returns a unique number to the client, and the client makes a request with this request number, and the server can complete de-interception.
However, in many scenarios, the request does not carry such a unique number! So can we use the requested parameters as a request identifier?
Consider a simple scenario first. Assuming that the request parameter has only one field, reqParam, we can use the following identifier to determine whether the request is repeated. User ID: interface name: request parameter
String KEY = "dedup:U="+userId + "M=" + method + "P=" + reqParam;
Then when the same user accesses the same interface and comes with the same reqParam, we can locate it as a duplicate.
But the problem is that our interface is usually not that simple. With the current mainstream, our parameter is usually a JSON. So for this kind of scene, how do we go about it?
Calculate the summary of request parameters as the parameter identifier
Suppose we sort the request parameters (JSON) in ascending order by KEY, and then combine them into a string as the KEY value? But this may be very long, so we can consider seeking a MD5 summary of this string as a parameter, and use this summary to replace the position of reqParam.
String KEY = "dedup:U="+userId + "M=" + method + "P=" + reqParamMD5;
In this way, the unique identifier of the request is marked!
Note: MD5 may be repeated theoretically, but the deduplication is usually within a short time window (for example, one second). The same user interface can spell different parameters in a short period of time, resulting in the same MD5. possible.
Continue to optimize, consider removing some time factors
The above problem is actually a very good solution, but when it is actually put into use, some problems may be found: some request users repeatedly click in a short time (for example, three requests are sent in 1000 milliseconds), but they are bypassed The above de-duplication judgment (different KEY value).
The reason is that there is a time field in the field of these request parameters. This field marks the time of the user's request, and the server can use this to discard some old requests (for example, 5 seconds ago). As in the following example, the other parameters of the request are the same, except that the request time differs by one second:
//The two requests are the same, but the request time is one second apart
String req = "{\n" +
"\"requestTime\" :\"20190101120001\",\n" +
"\"requestValue\" :\"1000\",\n" +
"\"requestKey\" :\"key\"\n" +
"}";
String req2 = "{\n" +
"\"requestTime\" :\"20190101120002\",\n" +
"\"requestValue\" :\"1000\",\n" +
"\"requestKey\" :\"key\"\n" +
"}";
For this kind of request, we may also need to block subsequent repeated requests. Therefore, it is necessary to eliminate such time fields before seeking business parameter summary. A similar field may be the latitude and longitude field of GPS (there may be very small differences between repeated requests).
Request deduplication tool class, Java implementation
public class ReqDedupHelper {
/**
*
* @param reqJSON request parameter, usually JSON here
* @param excludeKeys Which fields should be removed in the request parameters and then ask for a summary
* @return MD5 summary of parameters removed
*/
public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
String decreptParam = reqJSON;
TreeMap paramTreeMap = JSON.parseObject(decreptParam, TreeMap.class);
if (excludeKeys!=null) {
List<String> dedupExcludeKeys = Arrays.asList(excludeKeys);
if (!dedupExcludeKeys.isEmpty()) {
for (String dedupExcludeKey : dedupExcludeKeys) {
paramTreeMap.remove(dedupExcludeKey);
}
}
}
String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
String md5deDupParam = jdkMD5(paramTreeMapJSON);
log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
return md5deDupParam;
}
private static String jdkMD5(String src) {
String res = null;
try {
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
byte[] mdBytes = messageDigest.digest(src.getBytes());
res = DatatypeConverter.printHexBinary(mdBytes);
} catch (Exception e) {
log.error("",e);
}
return res;
}
}
Here are some test logs:
public static void main(String[] args) {
//The two requests are the same, but the request time is one second apart
String req = "{\n" +
"\"requestTime\" :\"20190101120001\",\n" +
"\"requestValue\" :\"1000\",\n" +
"\"requestKey\" :\"key\"\n" +
"}";
String req2 = "{\n" +
"\"requestTime\" :\"20190101120002\",\n" +
"\"requestValue\" :\"1000\",\n" +
"\"requestKey\" :\"key\"\n" +
"}";
//Compare all parameters, so the MD5 of the two parameters are different
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req);
String dedupMD52 = new ReqDedupHelper().dedupParamMD5(req2);
System.out.println("req1MD5 = "+ dedupMD5+" , req2MD5="+dedupMD52);
//Remove the time parameter comparison, MD5 is the same
String dedupMD53 = new ReqDedupHelper().dedupParamMD5(req,"requestTime");
String dedupMD54 = new ReqDedupHelper().dedupParamMD5(req2,"requestTime");
System.out.println("req1MD5 = "+ dedupMD53+" , req2MD5="+dedupMD54);
}
Log output:
req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267, req2MD5 = A2D20BAC78551C4CA09BEF97FE468A3F req1MD5 = C2A36FED15128E9E878583CAAAFEFDE9, req2MD5 = C2A36FED15128E9E878583CAAAFEFDE912
Log description:
- At the beginning, the two parameters are different because of the requestTime, so when seeking the deduplication parameter summary, you can find that the two values are different.
- In the second call, the requestTime was removed and the summary was requested ("requestTime" was passed in the second parameter), and it was found that the two summaries were the same, which was in line with expectations.
to sum up
So far, we can get a complete deduplication solution, as follows:
String userId= "12345678";//用户
String method = "pay";//接口名
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req,"requestTime");//计算请求参数摘要，其中剔除里面请求时间的干扰
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;
long expireTime = 1000;// 1000毫秒过期，1000ms内的重复请求会认为重复
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
// NOTE:直接SETNX不支持带过期时间，所以设置+过期不是原子操作，极端情况下可能设置了就不过期了，后面相同请求可能会误以为需要去重，所以这里使用底层API，保证SETNX+过期时间是原子操作
Boolean firstSet = stringRedisTemplate.execute((RedisCallback<Boolean>) connection -> connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime),
RedisStringCommands.SetOption.SET_IF_ABSENT));
final boolean isConsiderDup;
if (firstSet != null && firstSet) {
isConsiderDup = false;
} else {
isConsiderDup = true;
}
最后
感谢大家看到这里，文章有不足，欢迎大家指出；如果你觉得写得不错，那就给我一个赞吧。
也欢迎大家关注我的公众号：程序员麦冬，每天更新行业资讯！