Redis4.0 added a very useful lazy free characteristics, address the risk (mainly large collection of the specified element type Key) deleted Big Key fundamentally. The author redis operation and maintenance also come across a few Big Key to delete brought availability and performance problems.
This article is divided into the following sections describe redis lazy free:
- lazy free definition
- Why do we need lazy free
- Use the lazy free
- lazy free monitoring
- A simple analysis to achieve lazy free
lazy free definition
lazy free can be translated to delete an inert or delayed release; When the delete key, provides asynchronous delayed release Redis memory function key, the key release operation in bio (Background I / O) processing individual sub-thread, reduced deleted redis big key for blocking the main thread. Effectively avoid removing performance and availability problems caused by big key.
Why do we need lazy free
Redis is a single-thread program (except for a small amount of bio tasks), when running a large consuming request causes all queued requests redis not respond to other requests, causing performance problems, or even the occurrence of a failover cluster.
And when redis delete a large collection of keys, belong to this type of request time-consuming. By testing point of view, deleting a set of key elements of one million, it took about about 1000ms.
The following test, remove a 1 million hash key field, took 1360ms; handle this request DEL period, other requests are completely blocked.
删除一个100万字段的hash键
127.0.0.1:6379> HLEN hlazykey
(integer) 1000000
127.0.0.1:6379> del hlazykey
(integer) 1 (1.36s) 127.0.0.1:6379> SLOWLOG get 1) 1) (integer) 0 2) (integer) 1501314385 3) (integer) 1360908 4) 1) "del" 2) "hlazykey" 5) "127.0.0.1:35595" 6) “" Test estimation, reference may be; and hardware environment, Redis versions and load factors
| Key type | Item number | time consuming |
|---|---|---|
| Hash | ~ 1 million | ~1000ms |
| List | ~ 1 million | ~1000ms |
| Set | ~ 1 million | ~1000ms |
| Sorted Set | ~ 1 million | ~1000ms |
Before redis4.0, no lazy free function; DBA only through the tricky ways, similar to scan big key, delete every 100 elements; but in the face of "passive" scenario delete key, which can not do anything tricky deleted .
For example: We produce large clusters Redis Cluster, the business slowly written more than 20 million Hash key field with a TTL when the key expires, redis began to clean it when passive, resulting in redis blocked more than 20 seconds, the current The master node due to fragmentation than 20 seconds can not process the request, and the master database failover occurs.
After redis4.0 have lazy free features, such as active or passive When you delete a big key, and a O (1) time-consuming instruction, sub-millisecond return; the real release redis element takes action referred bio background tasks carried out.
Use the lazy free
lazy free use into two categories: The first category is automatically deleted corresponding to the DEL command, the second is expired key to delete, maxmemory key expel eliminated deleted.
Delete key initiative to use lazy free
UNLINK command
UNLINK command is the same as deleting the DEL key features of lazy free implementation.
The only not the same, UNLINK key when deleting collection classes, the number of elements is greater than if the set key 64 (described in detail later), will release operation real memory, to the individual bio operated.
Examples are as follows: Use UNLINK command to delete a large key mylist, which contains 2 million elements, but with only 0.03 milliseconds
127.0.0.1:7000> LLEN mylist
(integer) 2000000
127.0.0.1:7000> UNLINK mylist
(integer) 1 127.0.0.1:7000> SLOWLOG get 1) 1) (integer) 1 2) (integer) 1505465188 3) (integer) 30 4) 1) "UNLINK" 2) "mylist" 5) "127.0.0.1:17015" 6) "" Note: DEL command, blocking or deletion of concurrent
FLUSHALL/FLUSHDB ASYNC
By FLUSHALL / FLUSHDB When you add ASYNC asynchronous cleanup options, redis in cleaning up the entire instance or DB, operations are asynchronous.
127.0.0.1:7000> DBSIZE
(integer) 1812295
127.0.0.1:7000> flushall //同步清理实例数据，180万个key耗时1020毫秒
OK
(1.02s)
127.0.0.1:7000> DBSIZE
(integer) 1812637
127.0.0.1:7000> flushall async //异步清理实例数据，180万个key耗时约9毫秒
OK
127.0.0.1:7000> SLOWLOG get
1) 1) (integer) 2996109
2) (integer) 1505465989
3) (integer) 9274 //指令运行耗时9.2毫秒
4) 1) "flushall"
2) "async"
5) "127.0.0.1:20110"
6) ""
Delete key to use passive lazy free
lazy free passive deletion applied, there are four kinds of scenes, each scene corresponding to a configuration parameter; default is off.
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no slave-lazy-flush no Note: From test to see lazy free memory recovery efficiency is still relatively high; but in your production environment with the actual situation, open passive deleted
lazy free observation redis memory usage.
lazyfree-lazy-eviction
For redis memory usage reaches maxmeory, and when out of policy settings; when passive out of key, whether to adopt the lazy free mechanism;
because this scene turned lazy free, possible use out of key memory is not released in a timely manner, resulting in redis memory overdevelopment, over the restrictions maxmemory. When this scenario use, combined with operational testing.
lazyfree-lazy-expire --todo verification of this operation is synchronized to the DEL from the library or UNLINK.
There are key for setting the TTL, after it reaches its expiration, whether to adopt when redis clean up deleted lazy free mechanism;
this scenario is recommended to turn, because of the speed TTL itself is adaptively adjusted.
lazyfree-lazy-server-del
For some instructions, when handling key already exists, it will operate with a DEL key implicitly. As rename command , when the target key already exists, redis will delete the target key, if these targets are a key big key, it will delete the introduction of performance issues blocking. This parameter setting is to solve such problems, it is recommended to turn.
slave-lazy-flush
Carried out for the full amount of data synchronization slave, slave master before loading the RDB file, run flushall scene to clean up their own data,
parameter setting determines whether to adopt the abnormal flush mechanism. If the memory was little changed, it is recommended to turn. Full synchronization can reduce the amount of time-consuming, thereby reducing memory usage growth in the main library because the output buffer soared due.
lazy free monitoring
lazy free indicator can monitor data, only one value: lazyfree_pending_objects, represents redis perform lazy free operation, the actual number of the keys waiting to be recycled content. It does not reflect the large number of elements of a single key or wait recovered lazy free memory size.
This value is a certain reference value, the efficiency may be monitored or the number of stacked redis lazy free bond; such a small amount is deposited at flushall async scene.
A simple analysis to achieve lazy free
antirez lazy free to realize the function of many key functions and the underlying structure have been modified; this section only describes lazy free logic function implemented; code is mainly in the source and bio.c in lazyfree.c.
UNLINK command
unlink command entry function unlinkCommand () function call the same and del delGenericCommand () delete KEY operation, whether identified as lazyfree lazy invocation. If lazyfree, is called dbAsyncDelete () function.
But not necessarily every unlink command enable lazy free, redis KEY first determines the cost of the release (cost), performed only when the cost is greater than LAZYFREE_THRESHOLD lazy free.
Release cost calculation function key lazyfreeGetFreeEffort (), the type of set key, and satisfies the corresponding coding , the number of cost metadata key is set, otherwise cost is 1.
For example:
1 contains a list key 100 elements, its cost is Free 100
String key 2 a 512MB, its cost is a Free
can be see out, lazy free of the redis cost calculation complexity associated primary time.
lazyfreeGetFreeEffort () Function Code
size_t lazyfreeGetFreeEffort(robj *obj) { if (obj->type == OBJ_LIST) { quicklist *ql = obj->ptr; return ql->len; } else if (obj->type == OBJ_SET && obj->encoding == OBJ_ENCODING_HT) { dict *ht = obj->ptr; return dictSize(ht); } else if (obj->type == OBJ_ZSET && obj->encoding == OBJ_ENCODING_SKIPLIST){ zset *zs = obj->ptr; return zs->zsl->length; } else if (obj->type == OBJ_HASH && obj->encoding == OBJ_ENCODING_HT) { dict *ht = obj->ptr; return dictSize(ht); } else { return 1; /* Everything else is a single allocation. */ } } dbAsyncDelete () function code portion
#define LAZYFREE_THRESHOLD 64 //根据FREE一个key的cost是否大于64，用于判断是否进行lazy free调用
int dbAsyncDelete(redisDb *db, robj *key) { /* Deleting an entry from the expires dict will not free the sds of * the key, because it is shared with the main dictionary. */ if (dictSize(db->expires) > 0) dictDelete(db->expires,key->ptr); //从expires中直接删除key dictEntry *de = dictUnlink(db->dict,key->ptr); //进行unlink处理，但不进行实际free操作 if (de) { robj *val = dictGetVal(de); size_t free_effort = lazyfreeGetFreeEffort(val); //评估free当前key的代价 /* If releasing the object is too much work, let's put it into the * lazy free list. */ if (free_effort > LAZYFREE_THRESHOLD) { //如果free当前key cost>64, 则把它放在lazy free的list, 使用bio子线程进行实际free操作，不通过主线程运行 atomicIncr(lazyfree_objects,1); //待处理的lazyfree对象个数加1，通过info命令可查看 bioCreateBackgroundJob(BIO_LAZY_FREE,val,NULL,NULL); dictSetVal(db->dict,de,NULL); } } } The actual call lazyfreeFreeObjectFromBioThread in bio () function to release the key
void lazyfreeFreeObjectFromBioThread(robj *o) {
decrRefCount(o); //更新对应引用，根据不同类型，调用不同的free函数
atomicDecr(lazyfree_objects,1); //完成key的free,更新待处理lazyfree的键个数
}
flushall/flushdb async命令
When flushall / flushdb belt async, function emptyDb () call emptyDbAsync () to perform logical processing lazy free or entire instance DB.
emptyDbAsync processing logic is as follows:
/* Empty a Redis DB asynchronously. What the function does actually is to
* create a new empty set of hash tables and scheduling the old ones for
* lazy freeing. */
void emptyDbAsync(redisDb *db) { dict *oldht1 = db->dict, *oldht2 = db->expires; //把db的两个hash tables暂存起来 db->dict = dictCreate(&dbDictType,NULL); //为db创建两个空的hash tables db->expires = dictCreate(&keyptrDictType,NULL); atomicIncr(lazyfree_objects,dictSize(oldht1)); //更新待处理lazyfree的键个数，加上db的key个数 bioCreateBackgroundJob(BIO_LAZY_FREE,NULL,oldht1,oldht2);//加入到bio list } In the bio actual function call lazyfreeFreeDatabaseFromBioThread release db
void lazyfreeFreeDatabaseFromBioThread(dict *ht1, dict *ht2) {
size_t numkeys = dictSize(ht1);
dictRelease(ht1);
dictRelease(ht2);
atomicDecr(lazyfree_objects,numkeys);//完成整个DB的free,更新待处理lazyfree的键个数
}
Delete key to use passive lazy free
Remove four passive scenes, each scene Redis when called, the corresponding parameter is turned on is determined, if the parameter is on, is called lazy free function corresponding to the above processing logic implemented.
to sum up
Because the Redis is single main thread, antirez always stressed "Lazy Redis is better Redis".
The essence is to some lazy free of cost (mainly in-time copy of, occupy the main thread cpu time slice) higher deletion from the main redis stripped thread, so thread to handle bio child, greatly reducing the main line blocking time. Thereby reducing the deletion results in performance and stability problems.
Reference:
http://antirez.com/news/93