Redis principle - memory strategy

The address of the original text is updated, and the reading effect is better!

Redis Principle - Memory Strategy | CoderMast Programming Mast https://www.codermast.com/database/redis/redis-memery-strategy.html

Redis itself is a typical key-value memory storage database, so all keys and values ​​are stored in the previously learned Dict structure. However, in its database structure, there are two Dict: one is used to record key-value; the other is used to record key-TTL

typedef struct redisDb {
    dict *dict;     // 存放所有 key 和 value 的地方
    dict *expires;  // 存放每一个 key 及其对应的 TTL 存活时间，只包含设置了 TTL 的 key
    dict *blocking_keys;   
    dict *ready_keys;
    dict *watched_keys;
    int id;         // Database ID ：0 ~ 15
    long long avg_ttl;  
    unsigned long expires_cursor;   // expire 检查时在 dict 中抽样的索引位置
    list *defrag_later; // 等待碎片整理的 key 列表
} redisDb;

How does Redis know whether a key has expired?

Answer: Use two Dict to record key-value and key-ttl respectively

Is it deleted immediately when the TTL expires?

Answer: It is not deleted immediately when it expires, but  lazy deletion  and  periodic deletion are adopted .

#delete strategy

• Timed deletion: after the TTL expires, delete the corresponding key immediately

• Lazy deletion: It is not deleted immediately after the TTL expires, but when a key is accessed, the survival time of the key is checked, and the deletion is performed only if it has expired.

• Periodic deletion: It uses a scheduled task to periodically sample some expired keys and then delete them. There are two execution cycles:

  • Redis will set up a timing task serverCron() to perform expired key cleaning according to the frequency of server.hz, the mode is SLOW, and the default is 10
  • The beforeSleep() function will be called before each event cycle of Redis to perform expired key cleaning, and the mode is FAST

SLOW mode rules:

1. The execution frequency is affected by server.hz, and the default is 10, that is, it is executed 10 times per second, and each execution cycle is 100ms.

2. Executing cleanup takes no more than 25% of an execution cycle.

3. Traversing the db one by one, traversing the buckets in the db one by one, extracting 20 keys to determine whether they are expired

4. If the time limit (25ms) is not reached and the proportion of expired keys is greater than 10%, another sampling is performed, otherwise it ends

FAST mode rules (not executed if the proportion of expired keys is less than 10%)

1. The execution frequency is affected by the calling frequency of beforeSleep(), but the interval between two FAST modes is not less than 2ms
2. Performing cleanup takes no more than 1ms
3. Traversing the db one by one, traversing the buckets in the db one by one, extracting 20 keys to determine whether they are expired
4. If the time limit (1ms) is not reached and the proportion of expired keys is greater than 10%, another sampling is performed, otherwise it ends

#elimination strategy

Memory elimination is the process in which Redis actively selects and deletes some keys to release more memory when the memory usage of Redis reaches the set threshold. Redis will try to do memory elimination in the method processCommand() that processes client commands.

Redis supports a total of 8 elimination strategies

• noeviction: When the memory usage exceeds the configuration, an error will be returned, and no keys will be evicted.

• allkeys-lru: When adding a key, if the limit is exceeded, the key that has not been used for the longest time is first evicted through the LRU algorithm.

• volatile-lru: If the limit is exceeded when adding a key, the key that has not been used for the longest time is first evicted from the key set with an expiration time set.

• allkeys-random: If the limit is exceeded when adding a key, it will be randomly deleted from all keys.

• volatile-random: If the limit is exceeded when adding a key, it will be randomly evicted from the collection of expired keys.

• volatile-ttl: Evict keys that are about to expire from keys configured with an expiration time.

• volatile-lfu: Evict the least frequently used key from all keys configured with an expiration time.

• allkeys-lfu: Evict the least frequently used key from all keys.

LRU (Least Recently Used): The least recently used, the current time minus the last access time, the larger the value, the higher the elimination priority.

LFU (Least Frequently Used): The least frequently used, will count the access frequency of each key, the smaller the value, the higher the elimination priority.

typedef struct redisObject(

    unsigned type:4;//对象类型
    unsigned encoding:4;// 编码方式
    unsigned lru:LRU_BITS;
    //LRU: 以秒为单位记录最近一次访问时间，长度24bit
    //LFU: 高16位以分钟为单位记录最近一次访问时间，低8位记录逻辑访问次数
    int refcount;   // 引用计数，计数为0则可以回收
    void *ptr;      // 数据指针，指向真实数据
) robj;

The number of LRU accesses is called the number of logical accesses because it is not counted every time a key is accessed, but by calculation:

1. Generate a random number R between 0 and 1
2. Calculate 1 / (old times * lfu_log_factor + 1), log in R, lfu_log_factor defaults to 10
3. If R < P, then the counter + 1, and the maximum does not exceed 255
4. The number of visits will decrease with time, every lfu_decay_time minutes (default 1) from the last visit time, counter -1