As an important mechanism to periodically clean up invalid data, primary key invalidation exists in most cache systems, and Reids is no exception. Among the commands provided by Redis, EXPIRE, EXPIREAT, PEXPIRE, PEXPIREAT, SETEX and PSETEX can be used to set the expiration time of a Key-Value pair, and once a Key-Value pair is associated, the expiration time will expire It is automatically deleted (or more precisely becomes inaccessible). It can be said that the concept of primary key invalidation is relatively easy to understand, but how is it implemented in Redis? Recently, this blogger has raised several questions about the primary key failure mechanism in Redis, and has carefully explored it according to these questions. The summary is as follows, for the benefit of all spectators.
1. In addition to calling the PERSIST command, is there any other situation that will revoke the expiration time of a primary key? The answer is yes. First of all, when a primary key is deleted through the DEL command, the expiration time will naturally be revoked (isn't this nonsense, haha). Secondly, when a primary key with an expiration time set is updated and overwritten, the expiration time of the primary key will also be revoked (this seems to be nonsense, haha). But it should be noted that what is said here is that the primary key is updated and covered, not the value corresponding to the primary key is updated and covered, so SET, MSET or GETSET may cause the primary key to be updated. etc. are to update the value corresponding to the primary key. Such operations will not touch the expiration time of the primary key. In addition, there is a special command called RENAME. When we use RENAME to rename a primary key, the previously associated expiration time will be automatically passed to the new primary key, but if a primary key is overwritten by RENAME (such as primary key hello) It may be overwritten by the command RENAME world hello), at this time, the expiration time of the overridden primary key will be automatically revoked, and the new primary key will continue to maintain the characteristics of the original primary key.
2. How is the primary key invalidation in Redis implemented, that is, how is the invalid primary key deleted? In fact, there are two main ways for Redis to delete invalid primary keys: 1) passive way, if the primary key is found to be invalid when it is accessed, it will be deleted; 2) active way (active way), periodic Select a part of the expired primary keys from the primary keys with the expiration time to delete them. Next, we will explore the specific implementation of these two methods through code, but before that, let's take a look at how Redis manages and maintains the primary key (Note: The source code in this blog post is all from Redis-2.6. 12).
Code segment 1 gives the structure definition of the database in Redis. In this structure definition, except for the id, all are pointers to the dictionary. Among them, we only look at dict and expires. The former is used to maintain all the keys contained in a Redis database. -Value pair (its structure can be understood as dict[key]:value, that is, the mapping between primary key and value), the latter is used to maintain a primary key with expiration time set in a Redis database (its structure can be understood as expires[ key]:timeout, that is, the mapping between the primary key and the expiration time). When we use the SETEX and PSETEX commands to insert data into the system, Redis first adds the Key and Value to the dict dictionary table, and then adds the Key and expiration time to the expires dictionary table. When we use the
EXPIRE,
EXPIREAT,
PEXPIRE, and
PEXPIREAT commands to set the expiration time of a primary key, Redis first goes to the dictionary table dict to find out whether the primary key to be set exists, and if so, adds the primary key and expiration time to the expires dictionary table . In short, the primary key with the expiration time and the specific expiration time are all maintained in the expires dictionary table.
Code snippet one:
typedef struct redisDb {
dict *dict;
dict *expires;
dict *blocking_keys;
dict *ready_keys;
dict *watched_keys;
int id;
} redisDb;
After having a general understanding of how Redis maintains the primary key with the expiration time set, let's first take a look at how Redis implements passive deletion of the invalid primary key. Code segment 2 gives a function named expireIfNeeded, which will be called in any function that accesses data, that is to say, Redis will implement GET, MGET, HGET, LRANGE and other commands that involve reading data. Call it, the meaning of its existence is to check whether it is invalid before reading the data, and delete it if it is invalid. All relevant descriptions of the expireIfNeeded function are given in the code segment 2, and its implementation method will not be repeated here. What needs to be explained here is another function propagateExpire called in the expireIfNeeded function. This function is used to broadcast the information that the primary key has expired before the expired primary key is officially deleted. This information will be propagated to two destinations: one is sent to the AOF file , record the operation of deleting the invalid primary key in the standard command format of DEL Key; the other is all slaves sent to the current Redis server, and also inform these Slaves about the operation of deleting the invalid primary key in the standard command format of DEL Key Delete the respective invalid primary keys. From this, we can know that all Redis servers running as Slaves do not need to delete the invalid primary key through negative methods, they only need to obey the Master and it is OK!
Code segment two:
int expireIfNeeded(redisDb *db, robj *key) {
Get the expiration time of the primary key
long long when = getExpire(db,key);
If the expiration time is a negative number, it means that the primary key has no expiration time set (the expiration time defaults to -1), and returns 0 directly
if (when < 0) return 0;
If the Redis server is loading data from the RDB file, it does not delete the invalid primary key for the time being, and returns 0 directly
if (server.loading) return 0;
If the current Redis server is running as a Slave, then the deletion of the invalid primary key is not performed, because the Slave
The deletion of the invalid primary key is controlled by the Master, but here the invalidation time of the primary key will be compared with the current time.
Let's compare to inform the caller whether the primary key specified by the caller has expired
if (server.masterhost != NULL) {
return mstime() > when;
}
If none of the above conditions are met
, compare the expiration time of the primary key with the current time. If the specified primary key is found
Returns 0 if it has not expired
if (mstime() <= when) return 0;
If it is found that the primary key has indeed become invalid, first update the statistics about the invalid primary key, and then invalidate the primary key.
broadcast the valid information , and finally delete the primary key from the database
server.stat_expiredkeys++;
propagateExpire(db,key);
return dbDelete(db,key);
}
Code segment three:
void propagateExpire (redisDb * db, robj * key) {
robj *argv[2];
shared.del is a common Redis object that has been initialized at the beginning of the Redis server startup, that is, the DEL command
argv[0] = shared.del;
argv[1] = key;
incrRefCount(argv[0]);
incrRefCount(argv[1]);
Check whether AOF is enabled on the Redis server, and if so, record a DEL log for the invalid primary key
if (server.aof_state != REDIS_AOF_OFF)
feedAppendOnlyFile(server.delCommand,db->id,argv,2);
Check whether the Redis server has Slave, if so, send the command of DEL invalid primary key to all Slaves, this is
In the expireIfNeeded function above, it is no longer necessary to actively delete the invalid primary key when it is found that it is a Slave, because it
Just follow the command sent by the Master and it's OK
if (listLength(server.slaves))
replicationFeedSlaves(server.slaves,db->id,argv,2);
decrRefCount(argv[0]);
decrRefCount(argv[1]);
}
Above, we have learned how Redis deletes the invalid primary key in a negative way through the introduction of the expireIfNeeded function, but this method is obviously not enough, because if some invalid primary keys can't wait to be accessed again , Redis will never know that these primary keys are invalid, and will never delete them, which will undoubtedly lead to a waste of memory space. Therefore, Redis has also prepared a positive deletion method, which is implemented by using Redis' time events, that is, interrupting to complete some specified operations at regular intervals, including checking and deleting invalid primary keys. The callback function of the time event we are talking about here is serverCron, which is created when the Redis server starts. The number of executions per second is specified by the macro definition REDIS_DEFAULT_HZ, which is executed 10 times per second by default. Code segment four gives the program code when the time event is created, which is in the initServer function of the redis.c file. In fact, the serverCron callback function not only needs to check and delete the invalid primary key, but also update the statistics, control the client connection timeout, trigger BGSAVE and AOF, etc. Here we only focus on the implementation of deleting the invalid primary key. That is, the function activeExpireCycle.
Code segment four:
if(
aeCreateTimeEvent
(server.el, 1,
serverCron
, NULL, NULL) == AE_ERR) {
redisPanic("create time event failed");
exit(1);
}
Code segment five gives the implementation and detailed description of the function activeExpireCycle. The main implementation principle is to traverse the expires dictionary table of each database in the Redis server, and try to randomly sample REDIS_EXPIRELOOKUPS_PER_CRON (the default value is 10). Time primary keys, check whether they have expired and delete the invalid primary keys. If the number of invalid primary keys accounts for more than 25% of the sampling number, Redis will think that there are still many invalid primary keys in the current database, so it will continue Carry out the next round of random sampling and deletion, stop processing the current database until the ratio just now is lower than 25%, and turn to the next database. What we need to pay attention to here is that the activeExpireCycle function does not try to process all databases in Redis at one time, but only processes REDIS_DBCRON_DBS_PER_CALL (the default value is 16) at most. In addition, the activeExpireCycle function also has a processing time limit, not just how long it wants to execute. How long to execute, all these have only one purpose, that is to avoid the deletion of invalid primary keys from occupying too much CPU resources. Code segment five has a detailed description of all codes of activeExpireCycle, from which you can learn the specific implementation method of this function.
Code segment five:
void activeExpireCycle(void) {
Because each call to the activeExpireCycle function does not check all Redis databases at once, you need to record
The number of the last Redis database processed by each function call, so that the next time the activeExpireCycle function is called
It is also possible to continue processing from this database, which is why current_db is declared static, and another
The variable timelimit_exit is to record whether the execution time of the last call to
the activeExpireCycle function has reached
The time limit is reached, so it also needs to be declared as static
static unsigned int current_db = 0;
static int timelimit_exit = 0;
unsigned int j, iteration = 0;
The number of Redis databases processed each time the activeExpireCycle function is called is REDIS_DBCRON_DBS_PER_CALL
unsigned int dbs_per_call = REDIS_DBCRON_DBS_PER_CALL;
long long start = ustime(), timelimit;
If the number of databases in the current Redis server is less than REDIS_DBCRON_DBS_PER_CALL, all databases will be processed,
If the execution time of the last call to the activeExpireCycle function reaches the time limit, it means that there are many invalid primary keys, and
will choose to process all databases
if (dbs_per_call > server.dbnum || timelimit_exit)
dbs_per_call = server.dbnum;
Maximum time in microseconds to execute activeExpireCycle function, where
REDIS_EXPIRELOOKUPS_TIME_PERC
It is the proportion of CPU time that can be allocated to the
execution of the activeExpireCycle function per unit time, the default value is 25, server.hz
That is, the number of calls to activeExpireCycle in one second
, so this formula should be written more clearly like this, that is
(1000000 * (
REDIS_EXPIRELOOKUPS_TIME_PERC / 100)) /
server.hz
timelimit = 1000000*REDIS_EXPIRELOOKUPS_TIME_PERC/server.hz/100;
timelimit_exit = 0;
if (timelimit <= 0) timelimit = 1;
Traverse the invalid data in each Redis database
for (j = 0; j < dbs_per_call; j++) {
int expired;
redisDb *db = server.db+(current_db % server.dbnum);
Immediately add one to current_db here, so as to ensure that even if it is impossible to delete all current data within the time limit this time
The invalid primary key in the database will be processed from the next database the next time activeExpireCycle is called.
Thus
ensuring that each database has a chance to be processed
current_db++;
Start processing invalid primary keys in the current database
do {
unsigned long num, slots;
long long now;
If the size of the expires dictionary table is 0, it means that there is no primary key with expiration time set in the database, check directly
一数据库
if ((num = dictSize(db->expires)) == 0) break;
slots = dictSlots(db->expires);
now = mstime();
如果expires字典表不为空，但是其填充率不足1%，那么随机选择主键进行检查的代价
会很高，所以这里直接检查下一数据库
if (num && slots > DICT_HT_INITIAL_SIZE &&
(num*100/slots < 1)) break;
expired = 0;
如果expires字典表中的entry个数不足以达到抽样个数，则选择全部key作为抽样样本
if (num > REDIS_EXPIRELOOKUPS_PER_CRON)
num = REDIS_EXPIRELOOKUPS_PER_CRON;
while (num--) {
dictEntry *de;
long long t;
随机获取一个设置了失效时间的主键，检查其是否已经失效
if ((de = dictGetRandomKey(db->expires)) == NULL) break;
t = dictGetSignedIntegerVal(de);
if (now > t) {
发现该主键确实已经失效，删除该主键
sds key = dictGetKey(de);
robj *keyobj = createStringObject(key,sdslen(key));
同样要在删除前广播该主键的失效信息
propagateExpire(db,keyobj);
dbDelete(db,keyobj);
decrRefCount(keyobj);
expired++;
server.stat_expiredkeys++;
}
}
每进行一次抽样删除后对iteration加一，每16次抽样删除后检查本次执行时间是否
已经达到时间限制，如果已达到时间限制，则记录本次执行达到时间限制并退出
iteration++;
if ((iteration & 0xf) == 0 &&
(ustime()-start) > timelimit)
{
timelimit_exit = 1;
return;
}
如果失效的主键数占抽样数的百分比大于25%，则继续抽样删除过程
} while (expired > REDIS_EXPIRELOOKUPS_PER_CRON/4);
}
}
三、Memcached删除失效主键的方法与Redis有何异同？首先，Memcached在删除失效主键时也是采用的消极方法，即Memcached内部也不会监视主键是否失效，而是在通过Get访问主键时才会检查其是否已经失效。其次，Memcached与Redis在主键失效机制上的最大不同是，Memcached不会像Redis那样真正地去删除失效的主键，而只是简单地将失效主键占用的空间回收。这样当有新的数据写入到系统中时，Memcached会优先使用那些失效主键的空间。如果失效主键的空间用光了，Memcached还可以通过LRU机制来回收那些长期得不到访问的空间，因此Memcached并不需要像Redis中那样的周期性删除操作，这也是由Memcached使用的内存管理机制决定的。同时，这里需要指出的是Redis在出现OOM时同样可以通过配置maxmemory-policy这个参数来决定是否采用LRU机制来回收内存空间（感谢@Jonathan_Dai同学在博文
http://xenojoshua.com/2013/07/redis-lru/中对原文的指正
）！
）！
四、Redis的主键失效机制会不会影响系统性能？通过以上对Redis主键失效机制的介绍，我们知道虽然Redis会定期地检查设置了失效时间的主键并删除已经失效的主键，但是通过对每次处理数据库个数的限制、activeExpireCycle函数在一秒钟内执行次数的限制、分配给activeExpireCycle函数CPU时间的限制、继续删除主键的失效主键数百分比的限制，Redis已经大大降低了主键失效机制对系统整体性能的影响，但是如果在实际应用中出现大量主键在短时间内同时失效的情况还是会使得系统的响应能力降低，所以这种情况无疑应该避免。