The new feature of Redis4.0 - big KEY deletion

In Redis, the deletion of large keys has always been a headache. In order not to affect the service, we usually need to implement our own tools to delete large keys, or delete them during low business peaks.
In order to solve the above problems, Redis 4.0 newly added the UNLINK command to perform asynchronous deletion of large KEYs. So what is the logic behind this asynchronous delete?

Through the source code we can know the following information:
When we call the asynchronous delete UNLINK command:

Release Expire Dicti's reference to KV
Release Main Dict's reference to KV and record the Entry address of this KV
Calculate the cost of releasing this V, the calculation method is as follows:
3.1 If this V is a String type, the cost is 1
3.2 If this V is a composite type, the cost is the length of the composite type, for example, list is The result of llen, the hash is the result of hlen...
According to the obtained cost value, compare it with the cost threshold. If it is less than 64, the KV memory space can be released directly; if it is greater than 64, put the V into the lazyfree queue, and start a BIO background JOB to delete it.
4.1 In the background When a thread deletes V, it also performs different operations according to different types of V.
4.2 If it is a LIST type, it will release the space directly according to the length of the LIST.
4.3 If it is a SET type and the data structure is stored in a HASH table, then traverse the entire hash table and release the k and v spaces one by one; if the data structure is intset, just release the space directly
4.4 If it is a ZSET type, and the data structure adopts SKIPLIST For storage, since the bottom layer of SKIPLIST uses HASH + skiplist storage, the hash storage space in SKIPLIST will be released first, and then the skiplist part in SKIPLIST will be released; if the data structure is stored in ZIPLIST, the space will be released directly.
4.5 If it is HASH type and the data structure is stored in HASH table, then traverse the entire hash table and release k and v space one by one; if the data structure is stored in ZIPLIST, the space will be released directly.
Set V value equal to NULL
Free up KV space

The asynchronous deletion code is as follows:

int dbAsyncDelete(redisDb *db, robj *key) {
    /*  */
    if (dictSize(db->expires) > 0) dictDelete(db->expires,key->ptr); /* 在Main Dict 链表去掉引用，得到K-V entryDict */ dictEntry *de = dictUnlink(db->dict,key->ptr); if (de) { robj *val = dictGetVal(de); size_t free_effort = lazyfreeGetFreeEffort(val); /* 计算DEL key 的代价，根据代价决定是否采用异步删除方式 */ if (free_effort > LAZYFREE_THRESHOLD) { atomicIncr(lazyfree_objects,1,lazyfree_objects_mutex); bioCreateBackgroundJob(BIO_LAZY_FREE,val,NULL,NULL); dictSetVal(db->dict,de,NULL); } } /* 释放K-V空间，或者采用了异步删除方式，只需要释放Key空间 */ if (de) { dictFreeUnlinkedEntry(db->dict,de); if (server.cluster_enabled) slotToKeyDel(key); return 1; } else { return 0; } } /* 释放LIST 空间 */ void quicklistRelease(quicklist *quicklist) { unsigned long len; quicklistNode *current, *next; current = quicklist->head; len = quicklist->len; while (len--) { next = current->next; zfree(current->zl); quicklist->count -= current->count; zfree(current); quicklist->len--; current = next; } zfree(quicklist); } /* 释放HASH表空间 */ static int _dictClear(dict *ht) { unsigned long i; for (i = 0; i < ht->size && ht->used > 0; i++) { dictEntry *he, *nextHe; if ((he = ht->table[i]) == NULL) continue; while(he) { nextHe = he->next; dictFreeEntryKey(ht, he); dictFreeEntryVal(ht, he); free(he); ht->used--; he = nextHe; } } free(ht->table); _dictReset(ht); return DICT_OK; /* never fails */ }

Since asynchronous deletion actually removes the reference relationship between K and V in MAIN DICT, we cannot find it when we query the Key again, and then slowly release the memory space occupied by Value.

We found that when deleting asynchronously, whether it is deleting HASH or QUICKLIST, there is no speed control in this part, but a thread is started for him to delete, and it can run as fast as it can. This may cause the CPU to spike when we delete.

This deletion of the big KEY is carried out on the Master. What if this node has a Slave? What will the slave do? Also according to the code, it can be found that when we perform the UNLINK operation, we actually only send a DEL xxkey command when AOF and notify the slave. When the slave receives the del command, it will take the same judgment as above to delete the key.

notifyKeyspaceEvent(NOTIFY_GENERIC,"del",c->argv[j],c->db->id);

http://www.cnblogs.com/svan/p/7054129.html

The new feature of Redis4.0 - big KEY deletion

Guess you like