参考资料：Geek Time Redis Yafeng
Memory management
Let’s start with two questions:
1. How does Redis know whether a key has expired?
2 Will it be deleted immediately after the TTL expires?
Redis is a KV memory database, and all K and V are stored in Dict. However, there are 5 Dicts in its db structure. We only need to pay attention to two Dicts and Expires: one is used to record KV and the other is used to record K-TTL.
// 6.2的源码
typedef struct redisDb {
dict *dict; /* 数据存储dict, ALLKEYS*/
dict *expires; /* 上面的keys如果设置了过期时间则会存储到这里*/
dict *blocking_keys; /* Keys with clients waiting for data (BLPOP)*/
dict *ready_keys; /* Blocked keys that received a PUSH */
dict *watched_keys; /* WATCHED keys for MULTI/EXEC CAS */
int id; /* Database ID */
long long avg_ttl; /* Average TTL, just for stats */
unsigned long expires_cursor; /* Cursor of the active expire cycle. */
list *defrag_later; /* List of key names to attempt to defrag one by one, gradually. */
} redisDb;
How to really delete the Key?
1 Lazy deletion
When accessing a key, check the survival time of the key and delete it only if it expires. That is, when the user initiates a query, it will be checked and deleted if it expires.
robj *lookupKeyWriteWithFlags(redisDb *db, robj *key, int flags) {
expireIfNeeded(db,key); // 过期删除KEY
return lookupKey(db,key,flags);
}
2 Periodic deletion
Gu Mingsi suggested that through a scheduled task, he periodically samples partially expired keys and then deletes them. There are two execution cycles:
• Redis will set up a scheduled task serverCron() to perform expired key cleanup according to the frequency of server.hz, and the mode is SLOW
• Redis will call the beforesleep() function before each event loop , perform expired key cleanup, the mode is FAST
initServer. It is time to create a serverCron scheduled task.
/* Create the timer callback, this is our way to process many background
* operations incrementally, like clients timeout, eviction of unaccessed
* expired keys and so forth. */
if (aeCreateTimeEvent(server.el, 1, serverCron, NULL, NULL) == AE_ERR) {
serverPanic("Can't create event loop timers.");
exit(1);
}
int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
// 更新lruclock 到当前时间 为后期的LRU和LFU做准备
unsigned int lruclock = getLRUClock();
/* for debug purposes: skip actual cron work if pause_cron is on */
if (server.pause_cron) return 1000/server.hz;
/* We need to do a few operations on clients asynchronously. */
clientsCron();
/* 数据清理. */
databasesCron();
return 1000/server.hz; // 返回时间间隔 执行是时候和上一次返回值做比较
}
// databasesCron 的逻辑
void databasesCron(void) {
/* Expire keys by random sampling. Not required for slaves
* as master will synthesize DELs for us. */
if (server.active_expire_enabled) {
if (iAmMaster()) {
// 采用的是 Slow模式
activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW);
} else {
expireSlaveKeys();
}
}
}
// beforesleep 里面的逻辑
if (server.active_expire_enabled && server.masterhost == NULL)
activeExpireCycle(ACTIVE_EXPIRE_CYCLE_FAST); // Fast模式
SLOW mode rules:
1 The execution frequency is affected by server.hz, and the default is 10, that is, it is executed 10 times per second, and each execution period is 100ms.
2 The time required to perform cleanup does not exceed 25% of an execution cycle.
3 Traverse the db one by one, traverse the buckets in the db one by one, and extract 20 keys to determine whether they are expired.
4 If the time limit (25ms) is not reached and the proportion of expired keys is greater than 10%, perform another sampling, otherwise it ends.
FAST mode rules (not executed if the proportion of expired keys is less than 10%):
1 The execution frequency is affected by the frequency of beforeSleep() calls, but the interval between two FAST modes is not less than 2ms
2 The execution cleanup takes no more than 1ms
3 Traverse the db one by one, traverse the buckets in the db one by one, and extract 20 keys to determine whether they are expired4.
If the time limit (1ms) is not reached and the proportion of expired keys is greater than 10%, perform another sampling, otherwise the
总结
TTL record of RedisKey will be terminated. Method: Record the TTL time of each key in RedisDB through a Dict. Deletion strategy for expired keys:
• Lazy cleaning: Determine whether the key has expired every time you search for it, and delete it if it expires
• Periodic cleaning: Sample some keys regularly and judge Whether it has expired. If it expires, delete it.
There are two modes for regular cleaning:
• The execution frequency of SLOW mode is 10 by default, and each time does not exceed 25ms
. • The execution frequency of FAST mode is not fixed, but the interval between two times is not less than 2ms, and each time it takes no more than 1ms.
Memory elimination: This is the process in which Redis actively selects and deletes some keys to release more memory when the memory usage of Redis reaches the set threshold.
内存淘汰入口:
processCommand
if (server.maxmemory && !server.lua_timedout) {
int out_of_memory = (performEvictions() == EVICT_FAIL);
/* performEvictions may evict keys, so we need flush pending tracking
* invalidation keys. If we don't do this, we may get an invalidation
* message after we perform operation on the key, where in fact this
* message belongs to the old value of the key before it gets evicted.*/
trackingHandlePendingKeyInvalidations();
/* performEvictions may flush slave output buffers. This may result
* in a slave, that may be the active client, to be freed. */
if (server.current_client == NULL) return C_ERR;
int reject_cmd_on_oom = is_denyoom_command;
/* If client is in MULTI/EXEC context, queuing may consume an unlimited
* amount of memory, so we want to stop that.
* However, we never want to reject DISCARD, or even EXEC (unless it
* contains denied commands, in which case is_denyoom_command is already
* set. */
if (c->flags & CLIENT_MULTI &&
c->cmd->proc != execCommand &&
c->cmd->proc != discardCommand &&
c->cmd->proc != resetCommand) {
reject_cmd_on_oom = 1;
}
if (out_of_memory && reject_cmd_on_oom) {
rejectCommand(c, shared.oomerr);
return C_OK;
}
Redis supports 8 different strategies to select keys to delete:
noeviction: no keys will be eliminated, but new data will not be written when the memory is full. This is the default strategy.
volatile-ttl: For keys with TTL set, compare the remaining TTL value of the key. The smaller the TTL, the earlier it will be eliminated.
allkeys-random: All keys will be eliminated randomly. That is, volatile-random is selected directly from the dict
: keys with TTL set are randomly eliminated. That is, randomly selected from expires.
allkeys-lru: Eliminate all keys based on the LRU algorithm volatile
-lru: Eliminate keys with TTL based on the LRU algorithm
allkeys-lfu: Eliminate all keys based on the LFU algorithm volatile
-lfu: Yes After setting the TTL key,
there are two that are more likely to be confused for elimination based on the LFU algorithm :
LRU (Least Recently Used), least recently used. Subtract the last access time from the current time. The larger the value, the higher the elimination priority.
LFU (Least Frequently used), least frequently used. The access frequency of each key will be counted. The smaller the value, the higher the elimination priority.
typedef struct redisObject {
unsigned type:4;
unsigned encoding:4;
unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or
* LFU data (least significant 8 bits frequency
* and most significant 16 bits access time). */
// LRU 以毫秒为单位记录最近一次访问时间，长度24bit
// LFU 高16位以分钟为单位记录最近一次访问时间，低8位记录逻辑访问次数
int refcount; // 引用计数
void *ptr; // 数据指针 指向真实数据
} robj;
The number of accesses to LFU is called the number of logical accesses because it is not counted every time the key is accessed, but is generated through operation:
1 to generate a random number R between 0 and 1.
2 Calculate 1/(old times x Ifu_log_factor +1), recorded as P, lfu_log_factor defaults to 10.
3 If R < P, the counter + 1, and the maximum does not exceed 255.
4 The number of visits will decay over time. Every Ifu_decay_time minute (default 1) from the last visit time, the counter -1.
