Storage of hot data
Scenario:
There are 20 million data in the database, but only 1 million data in redis, how to ensure that all hot data is stored in redis?
Solution:
Limit the memory occupied by redis, and redis will leave hot data in memory according to its own data elimination strategy. So you can calculate the memory occupied by 100w data, and then set the redis memory limit, and set the elimination policy to allkeys-lru or volatile-lru. Set the
maximum memory occupied by redis: maxmemory 268435456
Set the expiration policy: maxmemory-policy volatile- lru
Statistics of active users in the past 24 hours (zset)
Such as user data. There are 2000w entries in the database. Use redis sortSet to put users who have logged in within two days (for the convenience of taking active users within a day), log in to ZADD once, and if the set already exists, its score (login time) will be overwritten. key: login:users, value: fraction timestamp, value userid. Set up a periodic task, such as deleting the data in the sort set before 3:00 of the previous day at 03:00:00 every day (to ensure that the set does not grow out of order, and keep active users within a day).
When fetching, get the current timestamp (int 10 digits), then subtract 1 day to get the active users in the past 24 hours according to the score range.
How to find the hot key
Method 1: Based on business experience, estimate which are hot keys.
In fact, this method is quite feasible. For example, if a certain product is doing flash sales, then the key of this product can be judged to be a hot key. The disadvantage is obvious, not all businesses can predict which keys are hot keys.
Method 2: Collect on the client side
This method is to add a line of code for data statistics before operating redis.
Method 3: For redis-cli
, if the elimination strategy is set to volatile-lfu or allkeys-lfu, you can use the following command to count hot keys
./redis-cli --hotkeys
–bigkeys is used to query big keys
Solutions for high-frequency access to hot keys
There are currently two options in the industry.
Leverage the second level cache
Use guavacache as a second-level cache.
There is a problem. The second-level cache of Guacache may be full. At this time, you can also set an elimination strategy for Guacache.
1. Based on the number of storage, the maximum number of values that can be stored. After exceeding, it will be clear according to the LRU algorithm.
2. Based on data access, set the expiration time, and it will expire after a long time without access.
3. Based on data update, set the expiration time,
and it will expire after a long time without updating. During gc, it will be recycled
backup hotkey
Store data in multiple redis nodes, and query from multiple redis nodes after query. For example, when I use a redis cluster cluster, at the code level, I add the IP of the machine after the key, and after hashing, after taking the modulo operation of 16384, I add it to the redis slot, which is relatively uniform. At this time, I can take it again data.