When Redis users should have experienced a large key deleted or expired, Redis server application scenarios appear blocked. This paper will be based on lazyfree inert deletion mechanisms Redis4.0 introduced, memory usage command to solve the obstruction caused by large delete key as well as the prevention of colorectal key generated accordingly introduced.
lazyfree mechanism
Lazyfree principle is carried out only when the deleted tombstone, the key release operation in bio (Background I / O) in the individual sub-threading, to reduce blocking key redis remove large main thread. Effectively avoid removing performance and availability problems caused by big key. The mention bio-threaded, single-threaded lot of people understand the Redis memory database, this is not entirely correct. Redis the main network transceiver and execute commands such as operations on the main thread work, but in addition there are several bio background thread, from source code can see a process of closing the file and background thread brush plate, and Redis4.0 newly added lazyfree thread.
/* Background job opcodes */
#define BIO_CLOSE_FILE 0 /* Deferred close(2) syscall. */
#define BIO_AOF_FSYNC 1 /* Deferred AOF fsync. */
#define BIO_LAZY_FREE 2 /* Deferred objects freeing. */
#define BIO_NUM_OPS 3
Now we come to understand lazyfree achieved through several command
unlink command
First look at the new command unlink
{"del",delCommand,-2,"w",0,NULL,1,-1,1,0,0},
{"unlink",unlinkCommand,-2,"wF",0,NULL,1,-1,1,0,0},
void delCommand(client *c) {
delGenericCommand(c,0);
}
void unlinkCommand(client *c) {
delGenericCommand(c,1);
}
This source can be seen by paragraphs del command and unlink commands are invoked delGenericCommand, the only difference is the second argument is not the same. This parameter is the asynchronous delete parameters.
/* This command implements DEL and LAZYDEL. */
void delGenericCommand(client *c, int lazy) {
numdel int = 0, j;
for (j = 1; j < c->argc; j++) {
expireIfNeeded(c->db,c->argv[j]);
int deleted = lazy ? dbAsyncDelete(c->db,c->argv[j]) :
dbSyncDelete(c->db,c->argv[j]);
if (deleted) {
signalModifiedKey(c->db,c->argv[j]);
notifyKeyspaceEvent(NOTIFY_GENERIC,
"del",c->argv[j],c->db->id);
server.dirty++;
numdel ++;
}
}
addReplyLongLong(c,numdel);
}
We can see delGenericCommand function to decide whether synchronous or asynchronous delete delete parameters according lazy. When performing unlink command, passing lazy parameter value 1, calls an asynchronous delete function dbAsyncDelete. Otherwise, the command execution del incoming parameter values 0, calling delete synchronization function dbSyncDelete. We look at focus delete achieve asynchronous logic dbAsyncDelete of.
/ * When the database have the delete key, or the value of the relevant entry expires, if there is enough memory to release the key, you can put it into a delayed release of the list to replace synchronous release. Inert delete the list of recovered bio.c in another thread. * /
#define LAZYFREE_THRESHOLD 64
/ * Define background delete threshold value, key elements is greater than the threshold value when the real threw a background thread to delete * /
int dbAsyncDelete(redisDb *db, robj *key) {
if (dictSize(db->expires) > 0) dictDelete(db->expires,key->ptr);
/*...*/
* = dictEntry of dictUnlink (db-> aforementioned key> ptr);
if (de) {
/ * LazyfreeGetFreeEffort to obtain the number of elements contained in the object val * /
robj * val = dictGetVal (de);
size_t free_effort = lazyfreeGetFreeEffort(val);
Deleted for background / * Delete key for determination of the threshold conditions are met * /
if (free_effort > LAZYFREE_THRESHOLD && val->refcount == 1) {
atomicIncr(lazyfree_objects,1);
bioCreateBackgroundJob(BIO_LAZY_FREE,val,NULL,NULL);
/ * Delete the object into BIO_LAZY_FREE background thread task queue * /
dictSetVal(db->dict,de,NULL);
/ * The first step to get the val value is set to null * /
}
}
/ * Delete the database dictionary entry, release resources * /
if (de) {
dictFreeUnlinkedEntry(db->dict,de);
if (server.cluster_enabled) slotToKeyDel(key);
return 1;
} else {
return 0;
}
}
The above mentioned delete key when threshold conditions are met, will key into BIO_LAZY_FREE background thread task queue. Next we look BIO_LAZY_FREE background thread.
/* Process the job accordingly to its type. */
if (type == BIO_CLOSE_FILE) {
close((long)job->arg1);
} else if (type == BIO_AOF_FSYNC) {
redis_fsync((long)job->arg1);
} else if (type == BIO_LAZY_FREE) {
/* What we free changes depending on what arguments are set:
* arg1 -> free the object at pointer.
* arg2 & arg3 -> free two dictionaries (a Redis DB).
* only arg3 -> free the skiplist. */
if (job->arg1)
lazyfreeFreeObjectFromBioThread(job->arg1);
else if (job->arg2 && job->arg3)
lazyfreeFreeDatabaseFromBioThread(job->arg2,job->arg3);
else if (job->arg3)
lazyfreeFreeSlotsMapFromBioThread(job->arg3);
} else {
serverPanic("Wrong job type in bioProcessBackgroundJobs().");
}
zfree(job);
I can see BIO_LAZY_FREE background thread to perform the corresponding function according to the job parameters
/ * Then delete the object function, reduce call decrRefCount key reference count, reference count is 0 will really release resources * /
void lazyfreeFreeObjectFromBioThread(robj *o) {
decrRefCount (a);
atomicDecr(lazyfree_objects,1);
}
/ * Background empty database dictionary, calling dictRelease loop through the database dictionary to remove all key * /
void lazyfreeFreeDatabaseFromBioThread(dict *ht1, dict *ht2) {
size_t numkeys = dictSize(ht1);
dictRelease(ht1);
dictRelease(ht2);
atomicDecr(lazyfree_objects,numkeys);
}
/ * Background delete key-slots mapping table in Redis cluster mode with * /
void lazyfreeFreeSlotsMapFromBioThread(rax *rt) {
size_t len = RT> name;
raxFree(rt);
atomicDecr(lazyfree_objects,len);
}
Unlink command logic can be summarized as follows: unlink call delGenericCommand perform the function passed parameter value 1 lazy to call the asynchronous delete function dbAsyncDelete, will meet the big key threshold into BIO_LAZY_FREE background thread task queue for asynchronous deleted.
Similar backstage delete command there flushdb async, flushall async, the principle is to obtain the deletion flag judge then calls an asynchronous delete function emptyDbAsnyc to clear the database specific implementation logic source code section to see flushdbCommand themselves, not be described herein.
In addition to active big key and delete empty erase data, the delete operation raises the expulsion expired key will be blocked Redis service. Therefore, in addition to increasing the Redis4.0 The above three commands to delete the background, but also an increase of four backstage delete configuration items, namely:
slave-lazy-flush: slave RDB After receiving option data files emptied; recommended open, slave node may be reduced flush operation time, thereby reducing the total amount of the primary synchronization time consuming.
lazyfree-lazy-eviction: the memory is full expelled option; this option may turn out of the memory release key is understaffed, leading to memory overdevelopment.
lazyfree-lazy-expire: Expiration key Delete option. It recommended that you turn.
lazyfree-lazy-server-del: delete internal options, such as when oldkey rename command to modify an existing newkey will be deleted first newkey. If newkey is a big key, it can cause blockage removed. It recommended that you turn.
The difference is not logical to remove the background, we are judged by the parameter option to choose whether to adopt dbAsyncDelete or emptyDbAsync asynchronous deleted.
memory usage
Blocking problems caused by large delete key can be solved by lazyfree, but the daily operation of large service Redis key causes blockage can not be avoided. Prevention is a big key to produce root of the problem.
Before the introduction of memory usage command in Redis4.0, mainly in the following two ways to find a Redis instance whether there is a big key:
1, redis-rdb-tools tools. Execution bgsave on redis instance, and then dump out rdb file for analysis, large KEY find them.
2, redis-cli --bigkeys command. You can find an example of five kinds of data types (String, hash, list, set, zset) the maximum key.
3, debug object key command. You can view the length of the sequence of a key, you can only find the information of a single key. The official is not recommended.
1, redis-rdb-tools tools.
For more information about rdb tools check out the links https://github.com/sripathikrishnan/redis-rdb-tools, this only describes memory-related use.
The basic command
rdb -c memory dump.rdb (wherein dump.rdb rdb is Redis instance document generated by bgsave).
For example # rdb -c memory dump.rdb
database,type,key,size_in_bytes,encoding,num_elements,len_largest_element
0,hash,hello1,1050,ziplist,86,22,
0,hash,hello2,2517,ziplist,222,8,
0,hash,hello3,2523,ziplist,156,12,
0,hash,hello4,62020,hashtable,776,32,
0,hash,hello5,71420,hashtable,1168,12,
You can see the information output includes data type, key, memory size, coding type and the like.
rdb tool advantage is to obtain detailed information on the key, optional parameters and more, support custom requirements. Csv json may generate or format to facilitate subsequent processing. Disadvantage is the need offline operation, a longer time get results.
2, redis-cli --bigkeys command
redis-cli --bigkeys is redis-cli comes with a command. It scans the entire redis looking for a large key, and print statistics.
E.g. redis-cli -p 6379 --bigkeys
# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type. You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).
[00.72%] Biggest hash found so far 'hello6' with 43 fields
[02.81%] Biggest string found so far 'hello7' with 31 bytes
[05.15%] Biggest string found so far 'hello8' with 32 bytes
[26.94%] Biggest hash found so far 'hello9' with 1795 fields
[32.00%] Biggest hash found so far 'hello10' with 4671 fields
[35.55%] Biggest string found so far 'hello11' with 36 bytes
-------- summary -------
Sampled 293070 keys in the keyspace!
Total key length in bytes is 8731143 (avg len 29.79)
Biggest string found 'hello11' has 36 bytes
Biggest hash found 'hello10' has 4671 fields
238027 strings with 2300436 bytes (81.22% of keys, avg size 9.66)
0 lists with 0 items (00.00% of keys, avg size 0.00)
0 sets with 0 members (00.00% of keys, avg size 0.00)
55043 hashs with 289965 fields (18.78% of keys, avg size 5.27)
0 zsets with 0 members (00.00% of keys, avg size 0.00)
You can see the print results are divided into two parts, part of the scanning process, only the current scan to stage the biggest key. summary section shows the data structure in each of Key largest and statistics.
Redis-cli --bigkeys advantage of an online scan, without blocking service. The disadvantage is that less information content is not precise enough.
Only results of the scan is the byte length of the string type metrics. list, set, zset, etc. are based on the number of elements as a yardstick, the number of elements and more will certainly not explain the memory for many.
In short, long before either off-line analytical methods when available, or is not detailed enough sample scan. From the ideal scanning online for more information at some distance. Redis4.0 introduced memory usage showed a superiority.
Now we come to understand memory usage characteristics from source.
{"memory",memoryCommand,-2,"rR",0,NULL,0,0,0,0,0}
void memoryCommand(client *c) {
/*...*/
/ * Key size is calculated to estimate the total size of the sample portion by the field. * /
else if (!strcasecmp(c->argv[1]->ptr,"usage") && c->argc >= 3) {
size_t usage = objectComputeSize(dictGetVal(de),samples);
use + = sdsAllocSize (dictGetKey (de));
usage += sizeof(dictEntry);
addReplyLongLong(c,usage);
}
}
Seen from the source memory usage is calculated by calling objectComputeSize key size. We look objectComputeSize logic function.
#define OBJ_COMPUTE_SIZE_DEF_SAMPLES 5 /* Default sample size. */
size_t objectComputeSize(robj *o, size_t sample_size) {
/ * ... code data types are classified, only to hash type instructions here * /
else if (o->type == OBJ_HASH) {
/ * Determines the type of hash coding mode, ziplist or Hashtable * /
if (o->encoding == OBJ_ENCODING_ZIPLIST) {
sit down = sizeof (* o) + (ziplistBlobLen (o-> ptr));
} else if (o->encoding == OBJ_ENCODING_HT) {
d = o->ptr;
= dictGetIterator of (d);
asize = sizeof(*o)+sizeof(dict)+(sizeof(struct dictEntry*)*dictSlots(d));
/ * A Field sampling cycle, the accumulation memory acquiring set sampling value, the default sampling of 5 samples * /
while((de = dictNext(di)) != NULL && samples < sample_size) {
he = dictGetKey (a);
ele2 = dictGetVal (de);
elesize + = sdsAllocSize (it) + sdsAllocSize (ele2);
elesize += sizeof(struct dictEntry);
samples++;
}
dictReleaseIterator (of);
/ * The sample memory value is divided by the sample on the sample size calculation step, and then multiplied by the total number of filed memory calculated total value * /
if (samples) asize += (double)elesize/samples*dictSize(d);
} else {
serverPanic("Unknown hash encoding");
}
Thereby, it is possible to know the default memory usage field sample 5 cycles accumulated to calculate the size of the entire memory of the key, the number of samples determines the accuracy of calculation cost and memory size of the key, the larger the sample, the more the number of cycles, the results more accurate, the more performance overhead.
We can scan Redis when the cluster via low peak Python script, with a smaller price to acquire all the memory size of the key. The following is pseudo-code portion, may be provided a large key warning threshold value according to the actual situation.
for key in r.scan_iter(count=1000):
redis-cli = '/usr/bin/redis-cli'
configcmd = '%s -h %s -p %s memory usage %s' % (redis-cli, rip,rport,key)
keymemory = commands.getoutput(configcmd)
to sum up
Redis4.0 and compared to the earlier version Redis5.0 not only solve the blocking problem before the big key delete lead, it also provides a memory garbage collection function after the big key to delete, avoid fragmentation rate is too high, take up more memory while reducing performance. In addition, it also provides an algorithm LFU expired, rdb & aof mixed persistent, dynamic HZ, a new type of data stream and other features, functionality and ease of use to enhance the display, but also is compatible with earlier versions of smooth migration, version upgrade is a good select.