I. Introduction
In Internet applications, caching has become a key component of high-concurrency architecture. This blog mainly introduces the typical scenarios of caching use, practical case analysis, Redis usage specifications and regular Redis monitoring.
Two, common cache comparison
Common caching solutions include local caching, including HashMap/ConcurrentHashMap, Ehcache, Memcache, Guava Cache, etc., and caching middleware including Redis, Tair, etc.
Three, Redis usage scenarios
1. counting
Redis implements fast counting and caching functions.
For example: the number of online viewers of a video or live broadcast will increase by 1 each time the user plays it.
2. Session centralized management
Session can be stored in the application service JVM, but this kind of solution will have consistency problems, and high concurrency will cause JVM memory overflow. Redis centrally manages the user's session. In this case, as long as the high availability and scalability of Redis are ensured, each user update or query login will directly obtain the information from Redis.
3. Speed limit
For example: the business requires that the user can only obtain the verification code 5 times within one minute.
4. Leaderboard
The query speed of relational databases is generally slow in rankings, so you can use redis's SortedSet to sort hot data.
For example, in a project, if you need to count the anchor’s gold ranking list, you can use the anchor’s id as the member, and the popularity value corresponding to the activity gift rewarded on that day as the score. You can get the anchor activity daily list through zrangebyscore.
5. Distributed lock
In actual multi-process concurrency scenarios, distributed locks are used to limit the concurrent execution of programs. It is mostly used to prevent cache breakdown in high concurrency scenarios.
Distributed locks are actually "occupying the pit". When another process executes setnx, it finds that the flag is already 1, and has to give up or wait.
Four, case analysis
1. [Case] Expiration setting-set command will remove the expiration time
All Redis data structures can be set to expire. If a string has an expiration time set, and then reset it, it will cause the expiration time to disappear. Therefore, in the project, it is necessary to reasonably evaluate the Redis capacity to avoid the frequent set that causes no expiration strategy, which indirectly causes the memory to be full.
The following is a screenshot of the Redis source code:
2. [Case] BUG about the expiration setting of Jedis 2.9.0 and below
It was found that Jedis had a bug when calling the expiredAt command. The pexpire command was finally called. This bug will cause the key to expire for a long time and cause Redis memory overflow and other problems. It is recommended to upgrade to Jedis 2.9.1 and above.
The source code of BinaryJedisCluster.java is as follows:
@Override
public Long pexpireAt(final byte[] key, final long millisecondsTimestamp) {
return new JedisClusterCommand<Long>(connectionHandler, maxAttempts) {
@Override
public Long execute(Jedis connection) {
return connection.pexpire(key, millisecondsTimestamp); //此处pexpire应该是pexpireAt
}
}.runBinary(key);
}Compare pexpire and pexpireAt:
For example, the current time we are using is 2018-06-14 17:00:00, and its unix timestamp is 1528966800000 milliseconds. When we use the PEXPIREAT command, because it is a past time, the corresponding key will expire immediately.
When we misuse the PEXPIRE command, the key will not expire immediately, but will expire after 1528966800000 milliseconds. The key expiration time will be quite long, about several W days later, which may cause Redis memory overflow, server crash and other problems.
3. [Case] Cache is penetrated
The cached key has an expiration strategy. If there are a large number of concurrent requests for this key at this point in time, these requests will usually return the source data from the back-end DB and reset to the cache when the cache expires. At this time, large concurrent requests may be Press down the back-end DB instantly.
There are two commonly used optimization solutions in the industry :
The first type: using distributed locks to ensure high concurrency, only one thread can return to the source back-end DB.
The second type: to ensure that the Redis key received by high concurrent requests is always valid, use non-user requests to return to the source backend, and change to active return to the source. Generally, asynchronous tasks can be used to actively refresh the cache.
4. [Case] Redis-standalone architecture prohibits the use of non-zero libraries
Redis executes the command select 0 and select 1 to switch, causing performance loss.
RedisTemplate will first obtain the link when executing the execute method.
Execute to RedisConnectionUtils.java, there will be a method to get the link.
JedisConnectionFactory.java will call the JedisConnection constructor. Note that the dbIndex here is the database number, such as: 1
Continue to follow up the JedisConnection code, when the selection library is greater than 1, there will be a select db operation. If you always use the 0 library, you don't need to execute additional cut library commands. Knowing the first place to cut select 1 from the library, where did select 0 come from?
In fact, the client uses Redis to release the link, but RedisTemplate has automatically released it for us, let us go back to the place where RedisTemplate executed the execute(...) method at the beginning.
The following is RedisConnectionUtils.java, which executes the code to close the link.
According to the code comments, if the selected library number is not 0, the spring-data-redis framework will reset select 0 every time!
In the vivo mall business, the performance of the product detail page interface has been optimized by more than 3 times.
Further verify that the database switch affects the performance at least about 3 times (depending on the specific business).
Rediscluster cluster database, default 0 database, can not choose other databases, which avoids this problem.
5. [Case] Beware of time complexity o(n) Redis command
Redis is single-threaded, so it is thread-safe.
Redis uses non-blocking IO, and the time complexity of most commands is O(1).
Using time-consuming commands is very dangerous. It will take up a lot of processing time in a single thread, causing all requests to be slowed down.
For example: Get all the elements smembers myset in the set collection, return all the members in the specified Hash, and the time complexity is O(N).
The value set in the cache becomes larger. When the high-parallel interface requests, the relevant data will be read from Redis. The reading time of each request becomes longer, and the continuous superimposition leads to the hot KEY situation, and a certain Redis fragment is blocked. The CPU usage reached 100%.
6. [Case] Cache hot key
In Redis, a key with a high access frequency is called a hot key. When a request for a hot key reaches the Server host, the amount of request is extremely large, which results in insufficient host resources and even downtime, affecting normal services.
There are two reasons for the hot key problem :
-
The data consumed by users is much larger than the data produced, such as hot or flash products, hot news, hot reviews, etc. These typical scenarios with more reading and less writing will cause hot issues.
- Request fragmentation is concentrated, exceeding the performance limit of a single server, such as a fixed name key, the hash falls into a server, and the amount of visits is extremely large. When the server limit is exceeded, the hot key problem will occur.
So in actual business, how to identify the hot key?
-
Based on business experience, estimate which are hot keys;
-
Client statistics collection, local statistics or report;
- If the server has an agent layer, it can be collected and reported at the agent layer;
When we recognize the hot key, how to solve the hot key problem ?
-
Redis cluster expansion: increase shard copies to balance read traffic;
-
Further hash the hot keys, such as backing up a key as key1, key2...keyN, N backups of the same data, N backups distributed to different shards, and random access to one of the N backups during access, further Share the read traffic.
- Use secondary cache, that is, local cache.
When a hot key is found, the data corresponding to the hot key is first loaded into the local cache of the application server to reduce read requests to Redis.
Five, Redis specification
1. It is forbidden to use non-database 0
Description:
The Redis-standalone architecture prohibits the use of other databases in Redis.
Reason:
Maintain compatibility for future business migration Redis Cluster.
When multiple databases are switched with select, it consumes more CPU resources.
It is easier to automate operation and maintenance management. For example, the scan/dbsize command is only used as a database.
- Some Redis Clients do not support single instance multiple databases due to thread safety issues.
2. Key design specification
Name the key prefix according to the business function to prevent key conflicts from being overwritten. It is recommended to use a colon to separate, for example, business name: table name: id:, such as live:rank:user:weekly:1:202003.
The length of the key is less than 30 characters, the key name itself is a String object, and Redis hard-codes the maximum length to 512MB.
In the Redis cache scenario, it is recommended to set the TTL value for all keys to ensure that the keys that are not used can be cleaned up or eliminated in time.
Key design prohibits the inclusion of special characters, such as spaces, line breaks, single and double quotation marks, and other escape characters.
3. Value design specification
The size of a single value must be controlled within 10KB, and the number of single-instance keys is too large, which may lead to delayed recovery of expired keys.
For complex data types such as set, hash, and list, the number of elements in the data structure should be reduced as much as possible, and it is recommended that the number should not exceed 1,000.
4. Pay attention to command time complexity
It is recommended to use O(1) commands, such as get scard.
The O(N) command focuses on the number of N. The following commands need to control the value of N at the business level.
hgetall
long
smembers
- zrange
For example: the time complexity of the smember command is O(n). When n continues to increase, it will cause the Redis CPU to continue to soar and block the execution of other commands;
5. Use of Pipeline
Description:
Pipeline is a way of Redis batch submission, that is, multiple command operations are established once and sent to Redis for execution, which will have better performance than circular single submission.
The Redis client executes a command in 4 processes: send command -> command queuing -> command execution -> return result.
The commonly used mget and mset commands effectively save RTT (command execution round-trip time), but hgetall does not have mhgetall, and batch operations are not supported. At this point, you need to use the Pipeline command
For example: in the live broadcast project, you need to query the anchor's daily, weekly, and monthly rankings at the same time, use PIPELINE to submit multiple commands at once, and return three ranking data at the same time.
6. Online disable command
- Prohibit the use of Monitor
It is forbidden to use the monitor command in the production environment. Under the condition of high concurrency, the monitor command will have the hidden danger of memory explosion and affecting the performance of Redis
- Prohibit the use of keys
The keys operation is to traverse all the keys. If there are too many keys, the slow query will block other commands. Therefore, the keys and keys pattern commands are prohibited.
It is recommended to use scan command online instead of keys command.
- Prohibit the use of Flushall, Flushdb
Delete all records in all databases in Redis, and the command is atomic and will not terminate execution. Once executed, it will not fail.
- Prohibit the use of Save
Block the current redis server until the persistence operation is completed, which will cause long-term blocking for instances with large memory.
- BGREWRITEAOF
Manual AOF, manual persistence will cause long-term blocking for instances with large memory.
- Config
Config is a client configuration method, which is not conducive to Redis operation and maintenance. It is recommended to set it in the Redis configuration file.
Six, Redis monitoring
1. Slow query
Method 1: slowlog to obtain slow query log
127.0.0.1:{port}> slowlog get 5
1) 1) (integer) 47
2) (integer) 1533810300
3) (integer) 175833
4) 1) "DEL"
2) "spring:session:expirations:1533810300000"
2) 1) (integer) 46
2) (integer) 1533810300
3) (integer) 117400
4) 1) "SMEMBERS"
Method 2: More comprehensive slow queries can be monitored through CacheCloud tools.
Path: "Application List"-click the relevant application name-click the "slow query" Tab page.
Click "Slow query" to focus on the number of slow queries and related commands.
2. Monitor the CPU core usage rate bound to the Redis instance
Since Redis is single-threaded, it focuses on monitoring the CPU core usage bound to the Redis instance.
Generally, the CPU resource utilization rate is about 10%. If the utilization rate is higher than 20%, consider whether to use RDB persistence.
3. Redis shard load balancing
The current redis-cluster architecture model, a cluster composed of 3 masters and 3 slaves, pay attention to the traffic balance of requests for each shard of Redis-cluster;
Obtain by command: redis-cli -p{port} -h{host} --stat
In general, an alarm is required for more than 12W.
4. Follow BigKey
Through the tools provided by Redis, redis-cli scans the corresponding Redis key regularly for optimization.
The specific command is as follows: redis-cli -h 127.0.0.1 -p {port} --bigkeys or redis-memory-for-key -s {IP} -p {port} XXX_KEY
Generally, more than 10K is a large key, which requires special attention. It is recommended to optimize from the business level.
5. Monitor the memory occupied by Redis
View the Info memory command to avoid performance problems caused by the exhaustion of the allocated MaxMemory under high concurrency scenarios.
Focus on the value corresponding to the used_memory_human configuration item. When the increment is too high, focus on evaluation is required.
Seven, summary
Combining specific business characteristics, reasonably evaluating the memory capacity required by Redis, selecting data types, and setting the single key size can better serve the business and provide high-performance guarantees for the business.
Author: Jessica Chen