Preface
Recently I am learning Redis related knowledge, and I read Ali's redis development specification and Redis development and operation and maintenance book. It is divided into four directions: usage specifications, pitted commands, actual project operations, and operation and maintenance configuration. I have sorted out 21 points of attention when using Redis. I hope it will be helpful to everyone. Let’s learn together.
Public account : The little boy picking up snails 
1. Redis usage specification
1.1, key specification points
When we design Redis keys, we should pay attention to the following points:
- Prefix the business name with the key and separate it with a colon to prevent key conflicts from being overwritten. For example, live:rank:1
- To ensure that the semantics of the key are clear, the length of the key should be less than 30 characters as much as possible.
- The key must not contain special characters, such as spaces, line breaks, single and double quotes, and other escape characters.
- Redis keys try to set ttl to ensure that keys that are not in use can be cleaned up or eliminated in time.
1.2, the main points of value specification
The value of Redis cannot be set arbitrarily.
The first point is that if a large number of bigKeys are stored, there will be problems, which will lead to slow queries, excessive memory growth, and so on.
- If it is a String type, the size of a single value is controlled within 10k.
- For hash, list, set, zset types, the number of elements generally does not exceed 5000.
The second point is to select the appropriate data type. Many small partners only use Redis's String type, which comes up with set and get. In fact, Redis provides a wealth of data structure types , and some business scenarios are more suitable for hash、zsetwaiting for other data results.
Counterexample:
set user:666:name jay
set user:666:age 18
复制代码Positive example
hmset user:666 name jay age 18
复制代码1.3. Set the expiration time for the Key, and pay attention to the keys of different businesses, and try to spread the expiration time a little
- Because Redis data is stored in memory, and memory resources are very precious.
- We generally use Redis as a cache, not a database , so the life cycle of the key should not be too long.
- Therefore, your key is generally recommended to use expire to set the expiration time .
If a large number of keys expire at a certain point in time, Redis may be stuck or even cache avalanche when it expires. Therefore, in general, the expiration time of keys for different businesses should be scattered. Sometimes, for the same business, you can also add a random value to the time to spread the expiration time.
1.4. It is recommended to use batch operations to improve efficiency
When we write SQL daily, we all know that batch operations will be more efficient. Updating 50 items at a time is more efficient than looping 50 times and updating one item each time. In fact, Redis operation commands are also the same.
A command executed by the Redis client can be divided into 4 processes: 1. Send command -> 2. Command queuing -> 3. Command execution -> 4. Return result. 1 and 4 are called RRT (command execution round trip time). Redis provides batch operation commands, such as mget, mset, etc., which can effectively save RRT. However, most commands do not support batch operations, such as hgetall, and mhgetall does not exist. Pipeline can solve this problem.
What is Pipeline? It can assemble a set of Redis commands, transmit them to Redis through an RTT, and then return the execution results of this set of Redis commands to the client in order.
Let's first look at the model that has executed n commands without using Pipeline:
Using Pipeline to execute n commands, the whole process requires 1 RTT, the model is as follows:
2. The commands that Redis has pitfalls
2.1. Caution O(n)complexity of the command, such as hgetall, smember, lrangeetc.
Because Redis executes commands in a single thread. The time complexity of commands such as hgetall and smember is O(n). When n continues to increase, the Redis CPU will continue to soar and block the execution of other commands.
Commands such as hgetall, smember, and lrange are not necessarily unavailable. It is necessary to comprehensively evaluate the amount of data, clarify the value of n, and then decide. For example, hgetall, if there are more hash elements n, hscan can be used first .
2.2 Use Redis's monitor command with caution
The Redis Monitor command is used to print out the commands received by the Redis server in real time. If we want to know what command operations the client has done to the redis server, we can use the Monitor command to view it, but it is generally used for debugging . Try not to use it in production. use! Because the monitor command may cause the memory of redis to continue to soar.
The monitor model is Jiangzi. It will output all the commands executed on the Redis server. Generally speaking, the QPS of the Redis server is very high, that is, if the monitor command is executed, the Redis server is in the output buffer of the Monitor client. There will be a lot of "inventory", which will take up a lot of Redis memory.
2.3. The keys command cannot be used in the production environment
The Redis Keys command is used to find all keys that match a given pattern. If you want to check how many keys of a certain type are in Redis, many friends think of using the keys command, as follows:
keys key前缀*
复制代码However, redis keystraverses matching, and the complexity is O（n）that the more data in the database, the slower. We know that redis is single-threaded. If there is a lot of data, the keys instruction will cause the redis thread to block, and the online service will also pause. The service will not resume until the instruction is executed. Therefore, generally in a production environment, do not use the keys command . The official document also states:
Warning: consider KEYS as a command that should only be used in production environments with extreme care. It may ruin performance when it is executed against large databases. This command is intended for debugging and special operations, such as changing your keyspace layout. Don't use KEYS in your regular application code. If you're looking for a way to find keys in a subset of your keyspace, consider using sets.
In fact, you can use the scan command, which provides pattern matching functions like the keys command. Its complexity is also O(n), but it is performed step by step through the cursor, and will not block the redis thread ; but there will be a certain probability of repetition , and it needs to be deduplicated once on the client .
scan supports incremental iterative commands, and incremental iterative commands also have disadvantages: for example, using the SMEMBERS command can return all elements currently contained in the set key, but for incremental iterative commands such as SCAN, because in During the incremental iteration of the keys, the keys may be modified, so the incremental iteration command can only provide limited guarantees for the returned elements.
2.4 Prohibit the use of fluxhall, flushdb
- The Flushall command is used to clear the data of the entire Redis server (delete all keys of all databases).
- The Flushdb command is used to clear all keys in the current database.
These two commands are atomic and will not terminate execution. Once the execution starts, the execution will not fail.
2.5 Pay attention to the use of del command
What command do you generally use to delete a key? Is it a direct del? If you delete a key, you can use the del command directly. But, have you thought about the time complexity of del? Let's discuss it by situation:
- If you delete a String type key, the time complexity is
O（1）, you can directly del . - If you delete a List/Hash/Set/ZSet type, its complexity is
O(n), n represents the number of elements.
Therefore, if you delete a List/Hash/Set/ZSet key, the more elements, the slower. When n is very large, pay special attention to it , which will block the main thread. So, if del is not used, how should we delete it?
- If it is a List type, you can execute
lpop或者rpopit until all elements are deleted.- If it is of Hash/Set/ZSet type, you can execute the
hscan/sscan/scanquery first , and then execute tohdel/srem/zremdelete each element in turn.
2.6 Avoid using SORT, SINTER and other highly complex commands.
Executing more complex commands will consume more CPU resources and block the main thread. So you should avoid executing such SORT、SINTER、SINTERSTORE、ZUNIONSTORE、ZINTERSTOREaggregation commands, it is generally recommended to put it on the client side for execution.
3. The actual operation of the project to avoid pits
3.1 Points to note when using distributed locks
Distributed lock is actually the realization of a lock that controls different processes of a distributed system to access shared resources together. Distributed locks are required for business scenarios such as placing orders with spikes and grabbing red envelopes. We often use Redis as a distributed lock, mainly with these points of attention:
3.1.1 The two commands SETNX + EXPIRE are written separately (a typical error implementation example)
if（jedis.setnx(key_resource_id,lock_value) == 1）{ //加锁
expire（key_resource_id，100）; //设置过期时间
try {
do something //业务请求
}catch(){
}
finally {
jedis.del(key_resource_id); //释放锁
}
}
复制代码If the setnxlock is executed and the expire time is about to be executed, the process crashes or needs to be restarted for maintenance, then the lock will be "immortal", and other threads will never be able to obtain the lock , so the general distributed lock cannot be like this achieve.
3.1.2 The SETNX + value value is the expiration time (some small partners implement this way, there are pits)
long expires = System.currentTimeMillis() + expireTime; //系统时间+设置的过期时间
String expiresStr = String.valueOf(expires);
// 如果当前锁不存在，返回加锁成功
if (jedis.setnx(key_resource_id, expiresStr) == 1) {
return true;
}
// 如果锁已经存在，获取锁的过期时间
String currentValueStr = jedis.get(key_resource_id);
// 如果获取到的过期时间，小于系统当前时间，表示已经过期
if (currentValueStr != null && Long.parseLong(currentValueStr) < System.currentTimeMillis()) {
// 锁已过期，获取上一个锁的过期时间，并设置现在锁的过期时间（不了解redis的getSet命令的小伙伴，可以去官网看下哈）
String oldValueStr = jedis.getSet(key_resource_id, expiresStr);
if (oldValueStr != null && oldValueStr.equals(currentValueStr)) {
// 考虑多线程并发的情况，只有一个线程的设置值和当前值相同，它才可以加锁
return true;
}
}
//其他情况，均返回加锁失败
return false;
}
复制代码Disadvantages of this scheme :
- The expiration time is generated by the client itself. In a distributed environment, the time of each client must be synchronized
- There is no unique identification of the holder, and it may be released/unlocked by other clients.
- When the lock expires, multiple concurrent clients request them at the same time, and they are all executed
jedis.getSet(). In the end, only one client can successfully lock the lock, but the expiration time of the client lock may be overwritten by other clients.
3.1.3: SET extended command (SET EX PX NX) (pay attention to possible problems)
if（jedis.set(key_resource_id, lock_value, "NX", "EX", 100s) == 1）{ //加锁
try {
do something //业务处理
}catch(){
}
finally {
jedis.del(key_resource_id); //释放锁
}
}
复制代码There may still be problems with this solution:
- The lock has expired and is released, and the business has not been executed yet.
- The lock was mistakenly deleted by another thread.
3.1.4 SET EX PX NX + verifies the unique random value and deletes it again (the problem of accidental deletion is solved, there is still the problem of lock expired and the business is not executed)
if（jedis.set(key_resource_id, uni_request_id, "NX", "EX", 100s) == 1）{ //加锁
try {
do something //业务处理
}catch(){
}
finally {
//判断是不是当前线程加的锁,是才释放
if (uni_request_id.equals(jedis.get(key_resource_id))) {
jedis.del(lockKey); //释放锁
}
}
}
复制代码Here, judging whether the lock added and released by the current thread is not an atomic operation. If you call jedis.del() to release the lock, the lock may no longer belong to the current client, and the lock added by others will be released.
Generally use lua script instead. The lua script is as follows:
if redis.call('get',KEYS[1]) == ARGV[1] then
return redis.call('del',KEYS[1])
else
return 0
end;
复制代码3.1.5 Redisson framework + Redlock algorithm solves the problem of lock expired release, business incomplete execution + stand-alone problem
Redisson uses a Watch dogsolution to solve the problem of lock expiration and release, and the business is not executed. The schematic diagram of Redisson is as follows:
The above distributed locks still have stand-alone problems: 
If thread one gets the lock on the Redis master node, but the locked key has not been synchronized to the slave node. At exactly this time, when the master node fails, a slave node will be upgraded to a master node. Thread two can acquire the lock of the same key, but thread one has already acquired the lock, and the security of the lock is lost.
For stand-alone problems, the Redlock algorithm can be used. Friends who are interested can read my article, seven solutions! Discuss the correct use posture of Redis distributed lock
3.2 Cautions for Cache Coherency
- If it is a read request, read the cache first, then read the database
- If a write request, update the database first, and then write to the cache
- Every time you update the data, you need to clear the cache
- Cache generally needs to set a certain expiration invalidation
- If the consistency requirements are high, biglog+MQ can be used to guarantee.
Friends who are interested, can read my article: In a concurrent environment, should you operate the database first or the cache first?
3.3 Properly evaluate the Redis capacity to avoid the invalidation of the previously set expiration time due to frequent set coverage.
We know that all the data structure types of Redis can set the expiration time. Suppose a string has an expiration time set, and if you reset it, the previous expiration time will be invalid.
The Redis setKeysource code is as follows:
void setKey(redisDb *db,robj *key,robj *val) {
if(lookupKeyWrite(db,key)==NULL) {
dbAdd(db,key,val);
}else{
dbOverwrite(db,key,val);
}
incrRefCount(val);
removeExpire(db,key); //去掉过期时间
signalModifiedKey(db,key);
}
复制代码In actual business development, at the same time, we must reasonably evaluate the capacity of Redis to avoid frequent set coverage, which will cause the key with the expiration time to become invalid. Novice Xiaobai is easy to make this mistake.
3.4 Cache penetration problem
Let's first look at a common way of using the cache: when a read request comes, check the cache first, and if the cache has a value hit, it will return directly; if the cache misses, it will check the database, then update the value of the database to the cache, and then return.
Cache penetration : refers to querying a certain non-existent data, because the cache is missed, it needs to be queried from the database, and if the data is not found, it will not be written to the cache, which will cause the non-existent data to go to the database every time a request is made Query, and then put pressure on the database.
In layman's terms, when a read request is accessed, neither the cache nor the database has a certain value, which will cause each query request for this value to penetrate the database, which is cache penetration.
Cache penetration is generally caused by these situations:
- Unreasonable business design , for example, most users do not have guards, but every request of yours is cached, and you can query whether a certain userid is guarded.
- Mistakes in business/operations/development , such as cache and database data have been deleted by mistake .
- Hacker illegal request attacks , such as hackers deliberately fabricating a large number of illegal requests to read non-existent business data.
How to avoid cache penetration? There are generally three methods.
-
- If it is an illegal request, we check the parameters at the API entrance to filter out illegal values.
-
- If the query database is empty, we can set a null value or default value for the cache. But if a write request comes in, the cache needs to be updated to ensure cache consistency, and at the same time, set an appropriate expiration time for the cache at the end. (Commonly used in business, simple and effective)
-
- Use bloom filters to quickly determine whether data exists. That is, when a query request comes in, the bloom filter is used to determine whether the value exists, and then the query continues.
Bloom filter principle: It consists of a bitmap array with an initial value of 0 and N hash functions. One performs N hash algorithms on a key to obtain N values, hashes these N values in the bit array and sets them to 1, and then when checking, if these specific positions are all 1, then bloom filtering The device determines that the key exists.
3.5 Cache Snow Ben problem
Caching Xueben: refers to the large amount of data in the cache until the expiration time, and the amount of query data is huge, and all requests directly access the database, causing excessive database pressure or even downtime.
- Cached snow ben is generally caused by the expiration of a large amount of data at the same time. For this reason, it can be solved by setting the expiration time evenly, that is, making the expiration time relatively discrete. If you use a larger fixed value + a smaller random value, 5 hours + 0 to 1800 seconds sauce purple.
- Redis failures and downtime may also cause the cache to run into snow. This requires the construction of a Redis high-availability cluster.
3.6 Cache breakdown problem
Cache breakdown: Refers to when the hot key expires at a certain point in time, and a large number of concurrent requests for this key happen to come at this point in time, and a large number of requests hit the db.
Cache breakdown looks a bit like, in fact, the difference between the two is that the cache Xueben means that the database is under too much pressure or even the machine is down. Cache breakdown is just a large number of concurrent requests to the DB database level. It can be considered that breakdown is a subset of the cache Xueben. Some articles believe that the difference between the two is that the breakdown is for a hot key cache, while Xueben has many keys.
There are two solutions:
- 1. Use the mutex lock scheme . When the cache fails, instead of loading the db data immediately, first use some atomic operation commands with a successful return, such as (Redis setnx) to operate, and then load the db database data and set the cache when it succeeds. Otherwise, try to get the cache again.
- 2. "Never expire" means that when the expiration time is not set, but the hot data is about to expire, the asynchronous thread updates and sets the expiration time.
3.7. Cache hot key problem
In Redis, we call the keys with high access frequency as hotspot keys. If a request for a hot key is sent to the server host, due to the extremely large amount of requests, the host may be insufficient in resources or even downtime, thereby affecting normal services.
And how is the hot key generated? There are two main reasons:
- The data consumed by users is much larger than the data produced, such as spikes, hot news, and other scenarios where read more and less write.
- Request fragmentation is concentrated, which exceeds the performance of a single Redi server. For example, with a fixed name key and Hash falling on the same server, the instantaneous traffic is extremely large, exceeding the machine bottleneck, and causing hot key issues.
So in daily development, how to identify the hot key?
- Judge which are hot keys based on experience;
- Client statistics report;
- Service agent layer reporting
How to solve the hot key problem?
- Redis cluster expansion: increase shard copies to balance read traffic;
- Hash the hot key, such as backing up a key as key1, key2...keyN, N backups of the same data, N backups distributed to different shards, and random access to one of the N backups during access. Further Share the read traffic;
- Use the second-level cache, the JVM local cache, to reduce Redis read requests.
4. Redis configuration operation and maintenance
4.1 Use long connections instead of short connections, and properly configure the client's connection pool
- If you use a short connection, you need to go through the TCP three-way handshake and four waved hands each time, which will increase time-consuming. However, for a long connection, it establishes a connection once, and redis commands can be used all the time. Jiangzi can reduce the time to establish a redis connection.
- The connection pool can establish multiple connections on the client without releasing them. When a connection is needed, there is no need to create a connection every time, which saves time. But you need to set the parameters reasonably, and you need to release the connection resources in time when you do not operate Redis for a long time.
4.2 Only use db0
The Redis-standalone architecture prohibits the use of non-db0. There are two reasons
- For a connection, Redis executes the command select 0 and select 1 to switch, which will consume new energy.
- Redis Cluster only supports db0, if you want to migrate, the cost is high
4.3 Set maxmemory + appropriate elimination strategy.
In order to prevent the memory backlog from swelling. For example, sometimes, the business volume has increased, the redis key is used a lot, the memory is directly insufficient, and the operation and maintenance brother also forgot to increase the memory. Does redis just hang up like this? Therefore, it is necessary to select the maxmemory-policy (maximum memory elimination strategy) and set the expiration time according to the actual business. There are a total of 8 memory elimination strategies:
- Volatile-lru: When the memory is insufficient to accommodate the newly written data, the LRU (least recently used) algorithm is used to eliminate the key with the expiration time;
- allkeys-lru: When the memory is insufficient to accommodate the newly written data, the LRU (least recently used) algorithm is used to eliminate all keys.
- Volatile-lfu: New in version 4.0. When the memory is insufficient to accommodate the newly written data, the LFU algorithm is used to delete the key from the expired key.
- allkeys-lfu: New in version 4.0, when the memory is not enough to accommodate the newly written data, the LFU algorithm is used to eliminate all keys;
- Volatile-random: When the memory is not enough to hold the newly written data, the data is randomly eliminated from the key with the expiration time set;.
- allkeys-random: When the memory is insufficient to accommodate the newly written data, data is randomly eliminated from all keys.
- Volatile-ttl: When the memory is not enough to accommodate the newly written data, among the keys with the expiration time set, the key will be eliminated according to the expiration time, and the earlier expired will be eliminated first;
- noeviction: The default strategy. When the memory is insufficient to accommodate the newly written data, the new write operation will report an error.
4.4 Enable lazy-free mechanism
Redis4.0+ version supports lazy-free mechanism. If your Redis still has bigKey, it is recommended to enable lazy-free. When it is turned on, if Redis deletes a bigkey, the time-consuming operation of releasing memory will be executed in the background thread, reducing the blocking impact on the main thread.
Author: snail boy picking
link: https: //juejin.cn/post/6942643266613411854
Source: Nuggets