background
We know that in interviews, interviewers often ask some questions about high concurrency. Among them, the hot key problem is a hot question. The hot key, as the name implies, is the key that is accessed very frequently. In addition to the high concurrency access conditions, what about the hot key cache problem? solve. Next, we analyze it from the following three aspects:
1. Hot key definition
2. How to find the hot key
3. Solution
Hot key definition
Suddenly there are hundreds of thousands or even larger requests to access a specific key on redis. This will cause the traffic to be too concentrated, reaching for example the Redis single instance bottleneck (usually 10W OPS level), or the upper limit of the physical network card, resulting in the Redis server being unable to hold. The consequence is that subsequent requests for this key will overwhelm the cache service
How to find hot keys
1. Based on business experience, predict those hot keys that are frequently accessed
Advantages: high feasibility. For example, in the spike product business, the spiked products are all hot keys
Disadvantages: Not all businesses are easy to predict hot keys, such as product recommendation business, you are unpredictable customer preferences in advance
2. Client collection
Advantages: The implementation is simple and convenient. You only need to add a line of code to perform data statistics before accessing the redis client. There are various statistical methods, including local counting, sending messages and processing statistics separately, etc.
Disadvantages: intrusion into client code
3. Collect at the Proxy (agent layer)
Advantages: the proxy layer has a unified entrance to do statistics, and there is no intrusion to the business code
Disadvantages: the cache architecture is required to have a proxy layer structure, for example, the proxy can be Twemproxy
4. Use redis's own commands
4.1, monitor command
The monitor command can capture the commands received by the redis server in real time, and then write code to calculate the hot key. Of course, there are also ready-made analysis tools for you to use, such as redis-faina.
Disadvantages: Under the condition of high concurrency, this command not only has the hidden danger of memory explosion, but also reduces the performance of redis
4.2, hotkeys parameters
Advantages: redis 4.0.3 provides the hot key discovery function of redis-cli, and the –hotkeys option can be added when redis-cli is executed, which is easy to operate
Disadvantages: When this parameter is executed, if there are more keys, the execution will be slower
5. Capture and analyze your own packets
The Redis client uses the TCP protocol to interact with the server, and the communication protocol uses RESP. Write your own program to monitor the port, parse the data according to the RESP protocol rules, and analyze it.
Disadvantages: high development cost, difficult maintenance, and possibility of packet loss
The above five schemes for discovering hot keys have their own advantages and disadvantages. You should choose the one that suits you best according to your business scenario.
solution
1. Use secondary cache (highly recommended)
For example, using guava-cache , or ehcache , or the most commonly used collection Hash, etc., after discovering the hot keys, load these hot keys into the JVM (either in the heap or off the heap) as a local cache. When accessing these keys, you can obtain them directly from the local cache, instead of directly accessing the redis layer, which effectively protects the cache server
2. Backup hot key
Don't let the same key be stored on the same redis machine. We store this key on multiple redis. When a hot key request comes in, we randomly select one on the backed up redis, access the value, and return the data.
Assuming that the number of redis clusters is N, the steps are shown in the following figure:
Disadvantages: The maintenance cost of the cache is very high. Assuming that there are 100 backup KEYs, then 100 KEYs need to be updated when deleting or updating, so this solution is not very recommended
Hot key expansion (industry solution)
After understanding the above discovery hot keys and solutions, we want to have a way to automatically discover the hot keys during the running of the project, and then the program will automatically process it? How is this done in the industry? There are actually only two steps:
1. Monitor hot key
2. Notify the system for processing
Business You Zan published a "You Zan Transparent Multi-Level Cache Solution (TMC)", which also mentioned hot key issues, we just take this to explain and learn
1), monitor hot key
The second method is praiseworthy: collect on the client side
In "Youzan Transparent Multi-Level Cache Solution (TMC)", there is a sentence mentioned:
TMC modified the JedisPool and Jedis classes of the native jedis package, and integrated the initialization logic of the Hermes-SDK package with the TMC " hot spot discovery" + "local cache " function during the initialization of the JedisPool.
Youzan rewritten the native jar package of jedis and added the Hermes-SDK package. What is the Hermes-SDK package used for? The answer is to do hot spot discovery and local caching. From a monitoring point of view, for each key value access request of the Jedis-Client, Hermes-SDK will asynchronously report the key access event to the Hermes server cluster through its communication module, so that it can perform "hot spot detection" based on the reported data. .
In addition to this method, some companies use method five in monitoring: capture and analyze by yourself
Specifically, this is to use flink to build a streaming computing system. Then write a packet capture program to capture the data of the redis listening port, and then discard the data in Kafka. Next, the streaming computing system consumes the data in Kafka and performs data statistics , which can also achieve the purpose of monitoring hot keys.
2), notify the system to do the processing
What is praiseworthy is the above solution one: use the second-level cache
After Youzan monitors the hot key, the Hermes server cluster will notify the Hermes-SDK in each business system through various means, telling them: "Brother, this key is a hot key, remember to cache it locally ." So Hermes-SDK The key will be cached locally for subsequent requests. Hermes-SDK finds that this is a hot key, which is taken directly from the local machine without accessing the cluster.
In addition to this notification method. We can also do the same. For example, if your streaming computing system detects a hot key, write it to a node in zookeeper. Then your business system monitors the node and finds that the node data has changed, which means that the hot key has been found. Finally, it is possible to write to the local cache .
to sum up
For the hot key problem, we only need to deal with it in two steps:
1. Monitor hot key
2. Notify the system for processing