The application that Xiaohei is responsible for is a management background application. The authority management uses the Shiro framework. Since there are multiple nodes, a distributed session is required, so Redis is used to store session information here.
Voiceover: I don’t know about distributed sessions, you can look at the four distributed consistent session implementation methods written by Xiaohei Ge.
Since Shiro did not directly provide Redis to store Session components, Xiao Hei had to use Github's open source component shiro-redis.
Since the Shiro framework needs to periodically verify whether the Session is valid, the bottom layer of Shiro will call SessionDAO#getActiveSessions to obtain all Session information.
Shiro-redis just inherits the SessionDAO interface, and the bottom layer uses the keys command to find all Session keys stored in Redis.
public Set<byte[]> keys(byte[] pattern){ checkAndInit(); Set<byte[]> keys = null; Jedis jedis = jedisPool.getResource(); try{ keys = jedis.keys(pattern); } Finally{ jedis.close(); } return keys; } Find the cause of the problem, the solution is relatively simple, find the solution on github, upgrade shiro-redis to the latest version.
In this version, shiro-redis uses scan command instead of keys to fix this problem.
public Set<byte[]> keys(byte[] pattern) {
Set<byte[]> keys = null;
Jedis jedis = jedisPool.getResource();
try{
keys = new HashSet<byte[]>();
ScanParams params = new ScanParams();
params.count(count);
params.match(pattern);
byte[] cursor = ScanParams.SCAN_POINTER_START_BINARY;
ScanResult<byte[]> scanResult;
do{
scanResult = jedis.scan(cursor,params);
keys.addAll(scanResult.getResult());
cursor = scanResult.getCursorAsBytes();
}while(scanResult.getStringCursor().compareTo(ScanParams.SCAN_POINTER_START) > 0);
}finally{
jedis.close();
}
return keys;
}
Although the problem was successfully solved, Brother Xiaohei was still a little puzzled.
Why does the keys command slow down the execution of other commands?
Why is the Keys command query so slow?
Why is there no problem with the Scan command?
How Redis executes commands
First, let’s look at the first question. Why does the keys command slow down the execution of other commands?
To answer this question, let's first look at the execution of a command on the Redis client:
From the perspective of the client, executing a command is divided into three steps:
Sending commands,
executing commands,
returning results,
but this is only the process that the client thinks, but in fact, there may be many clients sending commands to Redis at the same time, and Redis we all know that it uses a single-threaded model.
In order to process all client requests at the same time, Redis internally uses a queue method to queue for execution.
So the client actually needs four steps to execute a command:
Send command
Command queue
Execute command
Return result
Because Redis executes commands in a single thread, tasks can only be executed sequentially from the queue.
As long as the command execution speed in this process is too slow, other tasks in the queue have to wait. This looks to the external client as if Redis is blocked and no response is received.
Therefore, when using Redis, do not execute instructions that require long-running. This may cause Redis to block and affect the execution of other instructions.
Principle of KEYS
Next, I will answer the second question. Why is the query of Keys command so slow?
Before answering this question, please recall the underlying storage structure of Redis.
It’s okay if you don’t know your friends well. You can go back to the previous article "Ali Interviewer: HashMap familiar? Okay, let's talk about Redis dictionary!".
Here Xiao Hei Ge copied the content of the previous article. Redis uses a dictionary structure at the bottom layer, which is similar to the bottom layer of Java HashMap.
The keys command needs to return all Redis keys that match the given pattern. In order to achieve this goal, Redis has to traverse the underlying array of the ht[0] hash table in the dictionary. This time complexity is O(N) (N is Redis The total number of keys in the middle).
If the number of keys in Redis is small, the execution speed will still be fast. When the number of Redis keys gradually increases, rising to the level of millions, tens of millions, or even hundreds of millions, the execution speed will be very slow.
The following is an experiment done locally by Brother Xiaohei, using lua script to add 10W keys to Redis, and then using keys to query all keys. This query will probably block more than ten seconds.
eval "for i=1,100000 do redis.call('set',i,i+1) end"
0Hei brother here uses Docker to deploy Redis, the performance may be slightly worse.
SCAN Principle
Finally, let’s look at the third question. Why is there no problem with the scan command?
This is because the scan command uses a dark technology-cursor-based iterators.
Every time the scan command is called, Redis will return a new cursor and a certain number of keys to the user. Next time you want to continue to get the remaining keys, you need to pass this cursor to the scan command to continue the previous iteration process.
Simply put, the scan command uses paging to query redis.
The following is an example of the iterative process of the scan command:
The scan command uses cursors to ingeniously split a full query into multiple times, reducing query complexity.
Although the time complexity of the scan command is the same as that of keys, both are O(N), since the scan command only needs to return a small number of keys, the execution speed will be very fast.
Finally, although the scan command solves the shortage of keys, it also introduces other defects:
The same element may be returned multiple times, which requires our application to increase the ability to handle repeated elements.
If an element is added to redis during the iteration process, or deleted during the iteration process, that element will be returned, or it may not.
These defects need to be considered in our development.
In addition to scan, redis has several other commands for incremental iteration:
sscan: Used to iterate the database keys in the current database to solve the possible blocking problem of
smembers. The hscan command is used to iterate the key-value pairs in the hash key to solve the possible blocking problem of hgetall.
The zscan: command is used to iterate the elements in an ordered set (including element members and element scores), and is used to generate zrange, which may cause blocking problems.
Summary
Redis uses a single thread to execute operation commands. All clients send commands, and Redis will now put them in the queue, and then take out the corresponding commands from the queue in order.
If any task is executed too slowly, it will affect other tasks in the queue. In this way, from the perspective of the external client, the delay in getting a response from Redis will appear to be blocked.
Therefore, do not execute instructions that may cause blocking such as keys, smembers, hgetall, and zrange in production. If you really need to execute them, you can use the corresponding scan command to traverse progressively, which can effectively prevent blocking problems.
Absorbing material: www.goodsmaterial.com