This article requires a certain understanding of Redis common commands, underlying data structures, memory management, and clustering and other related concepts. It is recommended to view the summary of Redis interview questions to quickly learn related knowledge points.
1 Introduction
Presumably when you use Redis, you have encountered some "weird" scenes more or less, and there is a high probability of stepping on a "pit":
- A key with an expiration time set has not expired in the end;
- Using the command of
O(1)complexitySETBIT, it turned out to be OOM; - Use to read a key
RANDOMKEYrandomly , the main thread is blocked; - For the same command, the main library cannot find the data, but the slave library can find it;
- Why does the slave library use more memory than the main library;
- …
Being familiar with and understanding some common problems in Redis can help us quickly locate and solve problems. After sorting and sorting, the author roughly divides these problems into three parts:
- What are the pits of common commands?
- What are the pitfalls of data persistence?
- What pits are there in master-slave library synchronization?
2 Pit in common commands
Let's first look at some commonly known commands that may have "unexpected" results.
2.1 Accidental loss of expiration time
SETThe command must be the most used command in Redis. It can not only set the key-value, but also set the expiration time of the key, like the following:
127.0.0.1:6379> SET testkey val1 EX 60
OK
127.0.0.1:6379> TTL testkey
(integer) 59
However, it should be noted that if you modify the key from other parts of the code and do not add the "expiration time" parameter, the expiration time of the key will be "erased", that is, it will never expire .
127.0.0.1:6379> SET testkey val2
OK
127.0.0.1:6379> TTL testkey // key永远不过期了！
(integer) -1
If you find that the memory of Redis continues to grow, and many keys have an expiration time set originally, and later find that the expiration time is lost, it is likely to be caused by this reason.
2.2 DEL blocks Redis
When you need to delete a key, you may immediately think of the DELcommand , but have you thought about its time complexity?
When introducing the DEL command, the official Redis document describes it as follows: The time-consuming to delete a key is related to its type.
- The key is of type String, and the DEL time complexity is O(1);
- The key is of type List, Hash, Set, and ZSet, and the
DELtime complexity is O(M), where M is the number of elements.
If you understand the underlying data structure of Redis, it is not difficult to understand. Therefore, when deleting keys of the type List, Hash, Set, and ZSet , you must pay special attention to it. You cannot execute it without thinking DEL. Instead, you should delete ** in the following way:* *
1) Query the number of elements in the key: execute the LLEN HLEN SCARD ZCARDcommand ;
2) Determine the number of elements in the key: if it is less, it can be deleted directly, otherwise it can be deleted in batches;
3) Delete in batches : perform LRANGE/HSCAN/SSCAN/ZSCAN + LPOP/RPOP/HDEL/SREM/ZREM deletion.
The above is a collection type. If you delete a String type directly, will this not happen?
Obviously, it will also block the main thread of Redis. We know that a String type key can store up to 512M of data by default. When Redis releases such a large amount of memory to the operating system, it must be time-consuming.
lazy-freeDoes the mechanism not block the main thread if it is turned on?
Even if Redis turns on lazy-free, when deleting a bigkey of String type, it is still processed in the main thread instead of being executed in a background thread. Therefore, the risk of blocking Redis still cannot be solved.
2.3 RANDOMKEY blocks Redis
Redis provides a command to randomly view a key in memory: RANDOMKEY. This command will "randomly" take a key from Redis. Although it is called random , the execution speed is not as fast as we thought.
To explain this problem clearly, it is necessary to combine the expiration policy of Redis. Redis uses a combination of regular cleaning and lazy cleaning to clean up expired keys.
Let's take a look at RANDOMKEYthe execution process: After Redis randomly takes out a key, it will first check whether the key has expired. If the key has expired, then Redis will delete it, this process is lazy cleanup. But it can't end after cleaning, Redis has to repeat this cycle until a "non-expiring" key is found and returned to the client.
But here is a problem: if a large number of keys in Redis have expired at this time, but they have not been cleaned up in time, then this cycle will last for a long time to end, and this time is spent on cleaning up expired keys + Look for keys that do not expire. As a result, RANDOMKEYthe execution time becomes longer, which affects the performance of Redis.
The above process is mainly aimed at the Matser node. If the slave node executes the RANDOMKEYcommand , the problem will be more serious . The main reason is that Slave will not clean up expired keys by itself:
When a key is about to expire, the Master will clean up and delete it first, and then the Master will send a DELcommand tell the Slave to delete the key, so as to achieve data consistency between the master and slave libraries.
In the same scenario where there are a large number of keys that have expired but have not been cleaned up in the same Redis, the following problems will occur RANDOMKEYwhen :
1) Slave randomly takes out a key to determine whether it has expired;
2) The key has expired, but the Slave will not delete it, but will continue to randomly search for a key that does not expire;
3) Since a large number of keys have expired, the Slave will not be able to find a key that meets the conditions, and will fall into an "infinite loop" at this time.
That is to say, executing on Slave RANDOMKEYmay cause the entire Redis instance to be stuck.
This was actually a bug in Redis, and it wasn't fixed until 5.0. The solution given by Redis is :
When executing RANDOMKEYon , first determine whether all keys of the entire instance have an expiration time set. If so, in order to avoid finding a qualified key for a long time, the Slave will only search the hash table for at most 100 times, whether it can or not. If found, it will exit the loop. That is, a maximum number of retries is increased.
Therefore, if you find that Redis has "jitter" RANDOMKEYwhen , it is likely to be caused by this reason.
2.4 SETBIT causes OOM
In addition to storing a string of strings, the String type can often be used as a bitmap bitmap. In other words, split the value into individual bits to use:
127.0.0.1:6379> SETBIT testkey 10 1
(integer) 1
127.0.0.1:6379> GETBIT testkey 10
(integer) 1
But there is a pit: if the key does not exist, or the memory usage of the key is small, and the offset you want to operate is very large, then Redis needs to allocate a "larger memory space" , and the operation time will change. long, affecting performance.
Therefore, when you are SETBITusing , you must also pay attention to the size of the offset. Operating an offset that is too large will cause Redis to freeze. Of course, also pay attention to the impact of deletion.
2.5 MONITOR causes OOM
When you execute MONITORcommands , Redis will write each command to the client's "output buffer", and then the client will read the results returned by the server from this buffer.
However, if the QPS corresponding to Redis is high, this will cause the memory of the output buffer to continue to grow, occupying a lot of memory resources of Redis. If the memory resources of the machine happen to be insufficient, the Redis instance will face the risk of being OOM.