"Bite ......", a beautiful Saturday so was awakened by a burst of nails news.
Business group of students told me that many users are forced offline account today. Our normal logic system is the user login account after one, token valid time of day can be maintained. The problem is that user probably about every 10 minutes, you need to log in once again. This situation is typical for two reasons: 1, when the token is generated problems. 2, a problem verifying token.
By examining the log, I found the verification token, Redis has no corresponding token of. And to determine when to generate a new token, set the validity period to Redis is correct, then it is almost certain to be a problem in the Redis.
Then went to check the monitor Redis found at that time Redis memory footprint is too high due to force clean up several key. But from the point of view on the log, this time the situation did not appear skyrocketing traffic, and the number of key Redis did not significantly increase. That is what causes Redis memory footprint is too high it? After determining the Redis memory increase is not caused by us, we contacted the business group of students to help them, they said recently did on the line, and the new on-line features are to use Redis. But I still feel very strange, why Redis no increase in the key, and did not see other businesses key. After some inquiry, learned of the business group of students using the Redis of db1, and I used (and just investigation) is db0. Here is what I did there was negligence during troubleshooting.
It will affect each other in different db Redis do? Under normal circumstances, we use different db for data isolation, no question. But Redis clean up, clean up not only the amount of data that occupies the largest db, but will have to clean up all the db. Prior to this I do not really understand this knowledge, there are only guesses based on the phenomenon.
Curiosity drove me to verify this idea. So I decided to head straight to the source code Redis. Cleanup key related code in evict.c file.
Redis will be saved in a "Expired key pool", stored in this pool might be some clean-up key. Wherein the stored data is structured as follows:
struct evictionPoolEntry {
unsigned long long idle; /* Object idle time (inverse frequency for LFU) */
sds key; /* Key name. */
sds cached; /* Cached SDS object for key name. */
int dbid; /* Key DB number. */
};
Wherein the target idle time is idle in the Reids expired key algorithm, there are two: one is the LRU approximation, one is LFU. Default is approximately LRU.
Approximate LRU
Before explaining the approximate LRU, first take a brief look at the LRU. When the Redis memory footprint than maxmemory we set, will no longer use the key cleared away. According to LRU algorithm, we need all the key (also can be set to eliminate only have an expiration time key) sorted by idle time, idle time and then eliminated the largest portion of the data that Redis memory footprint down to a reasonable value .
LRU algorithm drawback is that we need to maintain a whole (or only the expiration time) a list of key, but also sorted by most recent use of time. This consumes a lot of memory, and is updated each time the key using the sort will take up additional CPU resources. For such high performance requirements Redis system is not allowed.
Thus, Redis approximation uses a LRU algorithm. When Redis receives a new write command, and memory are still not triggered approximate LRU algorithm to force clean up some of the key. Specific steps are clean, Redis key sampling will usually take five, then put the key into the expired we said above "Expired pool", key expiration pool is in accordance with the free time to sort, Redis will give priority to clean up idle the longest key, until the memory is less than maxmemory.
Approximate LRU algorithm cleaning results shown in Figure (images from Redis official documents)
That might be enough clear, we directly on the code.
Source code analysis
The figure shows the code in the main logic approximate LRU algorithm call path.
Wherein in the main logic freeMemoryIfNeededfunction
First, call the getMaxmemoryStatefunction to determine the current state of memory
int getMaxmemoryState(size_t *total, size_t *logical, size_t *tofree, float *level) {
size_t mem_reported, mem_used, mem_tofree;
mem_reported = zmalloc_used_memory();
if (total) *total = mem_reported;
int return_ok_asap = !server.maxmemory || mem_reported <= server.maxmemory;
if (return_ok_asap && !level) return C_OK;
mem_used = mem_reported;
size_t overhead = freeMemoryGetNotCountedMemory();
mem_used = (mem_used > overhead) ? mem_used-overhead : 0;
if (level) {
if (!server.maxmemory) {
*level = 0;
} else {
*level = (float)mem_used / (float)server.maxmemory;
}
}
if (return_ok_asap) return C_OK;
if (mem_used <= server.maxmemory) return C_OK;
mem_tofree = mem_used - server.maxmemory;
if (logical) *logical = mem_used;
if (tofree) *tofree = mem_tofree;
return C_ERR;
}
If you use a memory drops below maxmemory, then returns C_OK, otherwise it returns C_ERR. Further, this function also returns by transmitting some additional information pointer type parameters.
- Total : The total number of bytes that have been used, whether it is
C_OKstillC_ERRvalid. - the Logical : the used memory minus the size of the slave or AOF buffer, only to return
C_ERRvalid. - tofree : need to free memory size, and only returned
C_ERRvalid. - Level : Percentage of the memory used, generally between 0 and 1, when the memory limits, it is greater than 1. Whether it is
C_OKstillC_ERRvalid.
After completion of the memory state to determine if memory does not exceed the usage limits will be returned directly, otherwise it goes on. At this point we already know how much memory space to be released, the following will begin to release the memory of the operation. Each time the memory will be released to release the recording size of the memory until the memory is not less than the release tofree.
First, according maxmemory_policyto judge, there are different implementations for different purge policy, we look at the specific implementation of LRU.
for (i = 0; i < server.dbnum; i++) {
db = server.db+i;
dict = (server.maxmemory_policy & MAXMEMORY_FLAG_ALLKEYS) ?
db->dict : db->expires;
if ((keys = dictSize(dict)) != 0) {
evictionPoolPopulate(i, dict, db->dict, pool);
total_keys += keys;
}
}
The first is filled with "outdated pool", where traversing every db (validated my idea of the beginning), call the evictionPoolPopulatefunction to fill.
void evictionPoolPopulate(int dbid, dict *sampledict, dict *keydict, struct evictionPoolEntry *pool) {
int j, k, count;
dictEntry *samples[server.maxmemory_samples];
count = dictGetSomeKeys(sampledict,samples,server.maxmemory_samples);
for (j = 0; j < count; j++) {
unsigned long long idle;
sds key;
robj *o;
dictEntry *de;
de = samples[j];
key = dictGetKey(de);
/* some code */
if (server.maxmemory_policy & MAXMEMORY_FLAG_LRU) {
idle = estimateObjectIdleTime(o);
}
/* some code */
k = 0;
while (k < EVPOOL_SIZE &&
pool[k].key &&
pool[k].idle < idle) k++;
if (k == 0 && pool[EVPOOL_SIZE-1].key != NULL) {
continue;
} else if (k < EVPOOL_SIZE && pool[k].key == NULL) {
} else {
if (pool[EVPOOL_SIZE-1].key == NULL) {
sds cached = pool[EVPOOL_SIZE-1].cached;
memmove(pool+k+1,pool+k,
sizeof(pool[0])*(EVPOOL_SIZE-k-1));
pool[k].cached = cached;
} else {
k--;
sds cached = pool[0].cached; /* Save SDS before overwriting. */
if (pool[0].key != pool[0].cached) sdsfree(pool[0].key);
memmove(pool,pool+1,sizeof(pool[0])*k);
pool[k].cached = cached;
}
}
/* some code */
}
}
Due to space reasons, I intercepted a part of the code, we can see by this code, Redis is sampling the first part of the key, where the number of samples maxmemory_samples usually 5, we can also set up their own, the greater the number of samples, the results closer the results LRU algorithm, the impact of the performance will deteriorate.
After sampling we need to get every key free time, and then fill it to the specified location "Expired pool" of. Here "Expired pool" in accordance with the idle time is small to large order, that is, idle greatly key row on the far right.
After completion of the filling "Overdue pool" would get from back to front to clean up the most appropriate key.
/* Go backward from best to worst element to evict. */
for (k = EVPOOL_SIZE-1; k >= 0; k--) {
if (pool[k].key == NULL) continue;
bestdbid = pool[k].dbid;
if (server.maxmemory_policy & MAXMEMORY_FLAG_ALLKEYS) {
de = dictFind(server.db[pool[k].dbid].dict,
pool[k].key);
} else {
de = dictFind(server.db[pool[k].dbid].expires,
pool[k].key);
}
/* some code */
if (de) {
bestkey = dictGetKey(de);
break;
}
}
When you find the need to remove the key, on the need for synchronous / asynchronous cleanup depending on the setting clean-up.
if (server.lazyfree_lazy_eviction)
dbAsyncDelete(db,keyobj);
else
dbSyncDelete(db,keyobj)
Finally, note the size of the space in this clean-up, used in cycling conditions to determine whether to continue to clean up.
delta -= (long long) zmalloc_used_memory();
mem_freed += delta;
Clean-up
Finally, we look Redis support of several clean-up
- noeviction : do not continue to process write requests (DEL can continue processing).
- LRU-AllKeys : approximately all key LRU
- LRU-volatile : approximate LRU algorithm out of the set of key expiration time
- Random-AllKeys : random eliminate some of the key from the key in all
- Random-volatile : random out of all set an expiration time for the key
- ttl-volatile : the shortest part of the phase-out period key
Redis4.0 LFU began to support the strategy, and LRU similar, it is divided into two types:
- LFU-volatile : LFU algorithm using phase-out set an expiration time of key
- LFU-AllKeys : be eliminated from all the key, use the LFU
Written in the last
Now I know Redis in memory reaches the upper limit of what had been done thing. The future will not only check your db up when things go wrong.
On the subsequent processing of the accident, I first of all is to allow business students to roll back the code, and then let them use a single Redis, this business again similar problems will not affect our service the account, but also the whole sphere of influence It will become more manageable.