Background problem
In the business use redis process, there is an exception read timeout.
Troubleshooting
direct cause
Operation and maintenance inquiries redis slow query log and found that the abnormal time node, there redis slow query log, execute the command sadd took 1 second. But because redis is single-threaded applications, blocking the implementation of a single command will cause the queue waiting for other commands, resulting in read timeout.
Depth investigation - Why so slow it sadd
Sadd Why so slow? Check to see redis document complexity sadd operations is O (1), the actual use of the machine docker build redis tested using scripts sadd, until the order was more than 800W more than 100 milliseconds case occasionally. (See later test)
Redis built environment
Lazy line test in the machine, running from the use of docker redis application, as follows:
docker pull redis # 使用redis3.x版本docker run -itv ~/redis.conf:/redis.conf -p 32768:6379 --name myredis5 -d redis redis-server /redis.conf
Test Script
#coding=utf-8import timeimport redisimport random
r = redis.Redis(host='x.x.x.x', port=xxxx, decode_responses=True)
k = 'key4'tarr = []
st = time.clock()
st2 = time.clock()
r.sadd(k, 1) # 创建连接也会有耗时for i in range(1, 1600000):
t1 = time.clock() * 1000
rn = random.randint(100000000000, 20000000000000)
r.sadd(k, rn)
t2 = time.clock() * 1000
c = t2 - t1
tarr.append(str(c)) if c > 100: print i, cprint time.clock()
s = "\n".join(tarr)with open('./result.txt', 'w') as f:
f.write(s)
Test Results
When arriving 800W start sadd need occasional phenomenon of 100ms.
problem analysis
Query a lot of information when I saw redis del operational complexity is O (n), add more background here timeout, for example as follows:
Slow query log Time: 16 00.00 minutes 01 seconds, the command is sadd prefix_20180215, and the key has an expiration time.
See here the answer has been pretty clear, is not sadd triggered redis expired delete operation, and because of the complexity of the del command is O (n), the time spent on the deletion of expired data.
Test reproducibility
I in Range for (. 1, 1000000): T1 = time.clock () * 1000 RN = the random.randint (100000000000, 20,000,000,000,000) r.sadd (K, RN) T2 = time.clock () 1000 * C = T2 - T1 tarr.append (STR (C)) IF C> 100: Print I, C X = int (the time.time ()) X = 10 + # delay expires r.expire 10 per second (k, 10) while True : Y = the time.time () T1 = time.clock () * 1000 RN = the random.randint (. 1, 1000000000) r.sadd (K, RN) T2 = time.clock () * 1000 tarr.append (STR ( c)) if c> 100: # reproducing sadd slow query case print i, c if y> x + 5: # timeout being BREAK breakprint time.clock ()
Steps to Reproduce very simple,
-
To a key sadd on enough data (one million)
-
Key to set a relative expiration time.
-
Sadd command continued to call, call recording time.
-
The last observation redis slow query log.
As suspect as slow query log appeared SADD command, took 1 second.
Solutions and summary
Because of the complexity of operation for redis del keys are set O (n), so for a set of keys, is preferably provided by fragmentation, to avoid excessively large value of a single key.
In addition, redis4.0 delay has been deleted by the support configuration can be achieved by an asynchronous delete operation lazyfree_lazy_expire / azyfree_lazy_eviction / lazyfree_lazy_server_del, to avoid blocking asynchronous
Further reading
Finally, let's look at redis3.x and 4.x handle delete key source of it.
redis Three mechanisms have eliminated key, namely
-
del command
-
Passive phased out (delete request command when the corresponding key expired)
-
Initiative to remove (redis key initiative to eliminate recovered memory)
Let's take a look at the entry code redis3.x version of three elimination mechanism above.
del command - delCommand
void delCommand(client *c) {
int deleted = 0, j; for (j = 1; j < c->argc; j++) {
expireIfNeeded(c->db,c->argv[j]); if (dbDelete(c->db,c->argv[j])) {
signalModifiedKey(c->db,c->argv[j]);
notifyKeyspaceEvent(NOTIFY_GENERIC, "del",c->argv[j],c->db->id);
server.dirty++;
deleted++;
}
}
addReplyLongLong(c,deleted);
}
Process flow is quite simple, first check whether the key expired, then call dbDelete delete
Passive eliminated - expireIfNeeded
int expireIfNeeded(redisDb *db, robj *key) { mstime_t when = getExpire(db,key); //获取过期时间
mstime_t now; if (when < 0) return 0; /* No expire for this key */
/* Don't expire anything while loading. It will be done later. */
if (server.loading) return 0; /* If we are in the context of a Lua script, we claim that time is
* blocked to when the Lua script started. This way a key can expire
* only the first time it is accessed and not in the middle of the
* script execution, making propagation to slaves / AOF consistent.
* See issue #1525 on Github for more information. */
now = server.lua_caller ? server.lua_time_start : mstime(); // 过去当前时间
/* If we are running in the context of a slave, return ASAP:
* the slave key expiration is controlled by the master that will
* send us synthesized DEL operations for expired keys.
*
* Still we try to return the right information to the caller,
* that is, 0 if we think the key should be still valid, 1 if
* we think the key is expired at this time. */
if (server.masterhost != NULL) return now > when; /* Return when this key has not expired */
if (now <= when) return 0;
/* Delete the key */
server.stat_expiredkeys++;
propagateExpire(db,key); // 把过期时间传递出去（从库、AOF备份等）
notifyKeyspaceEvent(NOTIFY_EXPIRED,
"expired", key, db-> id); // key to changes occurring in db notification for the message to pass through the pubsub pubsub, can be used as monitoring performed redis
return dbDelete (db, key);
}
Initiative to eliminate - serverCron
server.c file
int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
/**
* sth not important
*/
...
/* We need to do a few operations on clients asynchronously. */
clientsCron(); /* Handle background operations on Redis databases. */
databasesCron(); /**
* sth not important
*/
...
server.cronloops++; return 1000/server.hz;
}/* This function handles 'background' operations we are required to do
* incrementally in Redis databases, such as active key expiring, resizing,
* rehashing. */void databasesCron(void) { /* Expire keys by random sampling. Not required for slaves
* as master will synthesize DELs for us. */
if (server.active_expire_enabled && server.masterhost == NULL)
activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW); /**
* sth not important
*/
}/* Try to expire a few timed out keys. The algorithm used is adaptive and
* will use few CPU cycles if there are few expiring keys, otherwise
* it will get more aggressive to avoid that too much memory is used by
* keys that can be removed from the keyspace.
*
* No more than CRON_DBS_PER_CALL databases are tested at every
* iteration.
*
* This kind of call is used when Redis detects that timelimit_exit is
* true, so there is more work to do, and we do it more incrementally from
* the beforeSleep() function of the event loop.
*
* Expire cycle type:
*
* If type is ACTIVE_EXPIRE_CYCLE_FAST the function will try to run a
* "fast" expire cycle that takes no longer than EXPIRE_FAST_CYCLE_DURATION
* microseconds, and is not repeated again before the same amount of time.
*
* If type is ACTIVE_EXPIRE_CYCLE_SLOW, that normal expire cycle is
* executed, where the time limit is a percentage of the REDIS_HZ period
* as specified by the REDIS_EXPIRELOOKUPS_TIME_PERC define. */void activeExpireCycle(int type) { int dbs_per_call = CRON_DBS_PER_CALL; /* We usually should test CRON_DBS_PER_CALL per iteration, with
* two exceptions:
*
* 1) Don't test more DBs than we have.
* 2) If last time we hit the time limit, we want to scan all DBs
* in this iteration, as there is work to do in some DB and we don't want
* expired keys to use memory for too much time. */
if (dbs_per_call > server.dbnum || timelimit_exit)
dbs_per_call = server.dbnum; //每次清理扫描的数据库数
/* We can use at max ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC percentage of CPU time
* per iteration. Since this function gets called with a frequency of
* server.hz times per second, the following is the max amount of
* microseconds we can spend in this function. */
timelimit = 1000000*ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC/server.hz/100;
timelimit_exit = 0; if (timelimit <= 0) timelimit = 1; if (type == ACTIVE_EXPIRE_CYCLE_FAST)
timelimit = ACTIVE_EXPIRE_CYCLE_FAST_DURATION; /* in microseconds. */
for (j = 0; j < dbs_per_call; j++) { int expired;
redisDb *db = server.db+(current_db % server.dbnum); /* Increment the DB now so we are sure if we run out of time
* in the current DB we'll restart from the next. This allows to
* distribute the time evenly across DBs. */
current_db++; /* Continue to expire if at the end of the cycle more than 25%
* of the keys were expired. */
// 如果有超过25%的键过期了则继续扫描
do { unsigned long num, slots; long long now, ttl_sum; int ttl_samples; /* If there is nothing to expire try next DB ASAP. */
if ((num = dictSize(db->expires)) == 0) { //当前没有需要过期的键
db->avg_ttl = 0; break;
}
slots = dictSlots(db->expires);
now = mstime(); /* When there are less than 1% filled slots getting random
* keys is expensive, so stop here waiting for better times...
* The dictionary will be resized asap. */
if (num && slots > DICT_HT_INITIAL_SIZE &&
(num*100/slots < 1)) break; /* The main collection cycle. Sample random keys among keys
* with an expire set, checking for expired ones. */
expired = 0;
ttl_sum = 0;
= 0 ttl_samples; IF (NUM> ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP)
NUM = ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP; // 3.2.11 20 times
the while (num--) {
dictEntry * de; Long Long TTL; IF ((de = dictGetRandomKey (db-> Expires)) == NULL) break; // get a random key
ttl = dictGetSignedIntegerVal(de)-now; if (activeExpireCycleTryExpire(db,de,now)) expired++; if (ttl > 0) { /* We want the average TTL of keys yet not expired. */
ttl_sum = TTL +;
ttl_samples ++;
}
} / **
* here are some deletion time control logic and other logic.
* /
IF (timelimit_exit) return; / * We do REPEAT The Cycle Not less Within last IF there are 25% of Keys
* found in The Current expired The DB * /.
} while (expired > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4); // 20次 / 4
}
}/* ======================= Cron: called every 100 ms ======================== *//* Helper function for the activeExpireCycle() function.
* This function will try to expire the key that is stored in the hash table
* entry 'de' of the 'expires' hash table of a Redis database.
*
* If the key is found to be expired, it is removed from the database and
* 1 is returned. Otherwise no operation is performed and 0 is returned.
*
* When a key is expired, server.stat_expiredkeys is incremented.
*
* The parameter 'now' is the current time in milliseconds as is passed
* to the function to avoid too many gettimeofday() syscalls. */int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) { long long t = dictGetSignedIntegerVal(de); if (now > t) {
sds key = dictGetKey(de);
robj *keyobj = createStringObject(key,sdslen(key));
propagateExpire(db,keyobj);
dbDelete(db,keyobj);
notifyKeyspaceEvent(NOTIFY_EXPIRED, "expired",keyobj,db->id);
decrRefCount(keyobj);
server.stat_expiredkeys++; return 1;
} else { return 0;
}
}
Initiative to remove the call path serverCron -> databasesCron -> activeExpireCycle -> activeExpireCycleTryExpire, we mainly look activeExpireCycleTryExpire.
Initiative to eliminate is to be deleted by random sampling, random algorithm is very simple, it is carried out by random, random the first slot, then a random list of a node on a slot in. In addition it will be to determine the number and frequency of scanning a db initiative to eliminate deleted based on the number and the length of time expired keys.
Incidentally, talk about, serverCron is redis periodic tasks, registered by a timer, databasesCron initiative to eliminate key in addition to, but also do rehash, resize and other things.
Low-level calls
Three mechanisms are different, but the underlying method is the same --dbDelete their call.
db.c file
/* Delete a key, value, and associated expiration entry if any, from the DB */int dbDelete(redisDb *db, robj *key) { /* Deleting an entry from the expires dict will not free the sds of
* the key, because it is shared with the main dictionary. */
if (dictSize(db->expires) > 0) dictDelete(db->expires,key->ptr); if (dictDelete(db->dict,key->ptr) == DICT_OK) { if (server.cluster_enabled) slotToKeyDel(key); return 1;
} else { return 0;
}
}
dict.c file
int dictDelete(dict *ht, const void *key) { return dictGenericDelete(ht,key,0);
}/* Search and remove an element */static int dictGenericDelete(dict *d, const void *key, int nofree)
{
unsigned int h, idx;
dictEntry *he, *prevHe;
int table; if (d->ht[0].size == 0) return DICT_ERR; /* d->ht[0].table is NULL */
if (dictIsRehashing(d)) _dictRehashStep(d);
h = dictHashKey(d, key); for (table = 0; table <= 1; table++) {
idx = h & d->ht[table].sizemask;
he = d->ht[table].table[idx];
prevHe = NULL; while(he) { if (key==he->key || dictCompareKeys(d, key, he->key)) { /* Unlink the element from the list */
if (prevHe)
prevHe->next = he->next; else
d->ht[table].table[idx] = he->next; if (!nofree) {
dictFreeKey(d, he);
dictFreeVal(d, he);
}
zfree(he);
d->ht[table].used--; return DICT_OK;
}
prevHe = he;
he = he->next;
} if (!dictIsRehashing(d)) break;
} return DICT_ERR; /* not found */}/* ------------------------------- Macros ------------------------------------*/#define dictFreeVal(d, entry) \
if ((d)->type->valDestructor) \
(d)->type->valDestructor((d)->privdata, (entry)->v.val)
server.c
/* Db->dict, keys are sds strings, vals are Redis objects. */dictType dbDictType = {
dictSdsHash, /* hash function */
NULL, /* key dup */
NULL, /* val dup */
dictSdsKeyCompare, /* key compare */
dictSdsDestructor, /* key destructor */
dictObjectDestructor /* val destructor */};void dictObjectDestructor(void *privdata, void *val){
DICT_NOTUSED(privdata); if (val == NULL) return; /* Values of swapped out keys as set to NULL */
decrRefCount(val);
}
object.c
void decrRefCount(robj *o) { if (o->refcount <= 0) serverPanic("decrRefCount against refcount <= 0"); if (o->refcount == 1) { switch(o->type) { case OBJ_STRING: freeStringObject(o); break; case OBJ_LIST: freeListObject(o); break; case OBJ_SET: freeSetObject(o); break; case OBJ_ZSET: freeZsetObject(o); break; case OBJ_HASH: freeHashObject(o); break; default: serverPanic("Unknown object type"); break;
}
zfree(o);
} else {
o->refcount--;
}
}
void freeSetObject(robj *o) { switch (o->encoding) { case OBJ_ENCODING_HT:
dictRelease((dict*) o->ptr); break; case OBJ_ENCODING_INTSET:
zfree(o->ptr); break; default:
serverPanic("Unknown set encoding type");
}
}
Delete the core can be seen in dictFreeVal years, corresponds to a macro, the macro calls the corresponding dictType of valDestructor, which is specified in dbDictType dictObjectDestructor function, corresponding deletion in decrRefCount (strictly speaking, by reference counting management lifecycle)
DecrRefCount within a corresponding release means for each data type, the method to release the set freeSetObject way we look. There are two types of data based on Set of two treatments, intset only need to release the pointer just fine, if the hash table dictRelease method is called.
dict.c
/* Clear & Release the hash table */void dictRelease(dict *d)
{
_dictClear(d,&d->ht[0],NULL);
_dictClear(d,&d->ht[1],NULL);
zfree(d);
}/* Destroy an entire dictionary */int _dictClear(dict *d, dictht *ht, void(callback)(void *)) {
unsigned long i; /* Free all the elements */
for (i = 0; i < ht->size && ht->used > 0; i++) {
dictEntry *he, *nextHe; if (callback && (i & 65535) == 0) callback(d->privdata); if ((he = ht->table[i]) == NULL) continue; while(he) {
nextHe = he->next;
dictFreeKey(d, he);
dictFreeVal(d, he);
zfree(he);
ht->used--;
he = nextHe;
}
} /* Free the table and the allocated cache structure */
zfree(ht->table); /* Re-initialize the table */
_dictReset(ht); return DICT_OK; /* never fails */}
At this point (dictClear method) we can see that this is an O (N) process, ht need to traverse each element and deleted, so are blocking redis risk exists. (Even the initiative to eliminate the mechanism)
This has been addressed in redis4.x series by deleting delay.