Redis study notes (chapter six): Database

Redis is written in C language database NoSql use herein, this would explain how the Redis database is stored? As well as some of the operations and related databases.

db array of all the databases are stored in Redis redis.h / redisServer structure as follows:

struct redisServer {
    ......

    // 数据库
    redisDb *db;

    ......
}

Redis default will create 16 databases, each independently of each other.

Switching Database

Each Redis client also has its own target database, by default, the target database Redis client database is 0. But the client can also switch the target database through the select command.

Inside the server, db client status attributes redisClient structure of the recording current target client database, as follows:

typedef struct redisClient {

    // 套接字描述符
    int fd;

    // 当前正在使用的数据库
    redisDb *db;

    // 当前正在使用的数据库的 id （号码）
    int dictid;

    // 客户端的名字
    robj *name;             /* As set by CLIENT SETNAME */

} redisClient;

If a client's database to target No. 1 database, then the relationship between the client and the corresponding server client state status as follows (from "Design and Implementation of Redis second edition of" Chapter 9: Database):

Note: So far, Redis still not returned to the client command target database end, so try not to use multiple databases in the project, so as to avoid confusion.

Database key space

Redis is a key to the database server, database server, each by a redis.h / redisDb structure represented by the specific structure is as follows:

typedef struct redisDb {

    // 数据库键空间，保存着数据库中的所有键值对
    dict *dict;                 /* The keyspace for this DB */

    // 键的过期时间，字典的键为键，字典的值为过期事件 UNIX 时间戳
    dict *expires;              /* Timeout of keys with a timeout set */

    // 正处于阻塞状态的键
    dict *blocking_keys;        /* Keys with clients waiting for data (BLPOP) */

    // 可以解除阻塞的键
    dict *ready_keys;           /* Blocked keys that received a PUSH */

    // 正在被 WATCH 命令监视的键
    dict *watched_keys;         /* WATCHED keys for MULTI/EXEC CAS */

    struct evictionPoolEntry *eviction_pool;    /* Eviction pool of keys */

    // 数据库号码
    int id;                     /* Database ID */

    // 数据库的键的平均 TTL ，统计信息
    long long avg_ttl;          /* Average TTL, just for stats */

} redisDb;

Space key (db properties) and visible to the user database correspond directly to:

• The key is the space key database of keys, each key is a string object.
• I.e. the value of the key space database values, each value can be a string, lists, hash table object, and any one of a collection of objects Redis objects ordered set of objects.

And in the database to add, modify, db dictionaries delete keys are also operating.

Expiration policy key

Redis key expiration time set in the four writing:

• expire key t1: shows a key to key lifetime t1 seconds.
• pexpire key t1: shows a key lifetime setting key is t1 ms.
• expireat key t1: shows a key to key expiration time t1 to the time stamp specified number of seconds.
• pexpireat key t1: shows a key to key expiration time t1 to the time stamp specified number of milliseconds.

Although there are four different wording, but they do is one thing, it can be pumped into a unified approach. In fact Redis also did just that, expire, pexpire, expireat three commands are implemented using pexpireat command.

How to store Redis key expiration time of it?

expires dictionary redisDb structure stored in the database expiration time of all keys, we call this dictionary to expire dictionary:

• Expired dictionary key is a pointer that points to a key object key space (that is, a database key).
• Expired dictionary is an integer value type long long, the expiration time of the integer stored database of key points to a key (a millisecond accuracy UNIX timestamp).

How Redis key to remove the expiration time of it?

Command is: persist key

Redis database to do the operation is only to delete the corresponding dictionary expires in pairs.

How to judge a Redis key has expired it?

By dictionary expires, the program can check whether a given key is expired by the following steps:

1. Checks if the given key exists in the dictionary expired: If there is, then get the key expiration time.
2. Check whether the current UNIX time kill is greater than the key expiration time: If so, then the key has expired; otherwise, the key is not expired.

Redis exactly how to delete an expired key it?

We know that the expiration time database keys are stored in the dictionary expired, they know how to judge whether a key expired, the remaining question is based on expiration time: if a key has expired, it will be deleted when it?

This question has three possible answers, which represent three different deletion policy:

• Regularly delete: key expiration time set at the same time, create a timer (timer). Let the timer expires key time comes, in addition to the implementation of the key side immediately.
• Inert Delete: faire key expires regardless, but each time get a key from the key space are made to check whether the key expires, if expired, then, in addition to the side of the key; if not expired, return the key.
• Deleted regularly: Every so often, the program checks the database once, delete the expired keys inside. As for how much you want to remove expired keys, as well as to check the number of databases, the decision by the algorithm.

In these three strategies, the first and third to active deletion policy, and the second was in addition to the passive side policies. Regardless of what kind of strategy has its advantages and disadvantages.

For regularly delete it:

• The advantage is you can guarantee expired keys will be deleted as soon as possible, and release the memory occupied key expired;
• The disadvantage is that will occupy part of the CPU time, especially when a lot of key time, CPU time will increase, which is intolerable.

For inert delete it:

• Taking out only the program key is checked, so the advantage is almost no CPU time;
• The disadvantage is that can cause memory leaks, such as when the key expires never visit this time is a memory leak.

For periodically delete it, it is an integration of these two strategies, periodically delete outdated policy enforcement delete key operations from time to time, and delete operations to reduce the impact on CPU time by limiting the duration and frequency of the deletion operations performed ; in addition, by periodically delete expired key, periodically delete the policy effectively reduces because of expired keys brought wasted memory.
Periodically delete the policy challenge is to determine the length of time to perform the delete operation and frequency:

• If you remove performed too frequently or too long time to execute, periodically delete policy will degenerate into a timing to delete the policy, so that the CPU time on the excess expired key operation other side in the above.
• If the delete operation to perform too little time or too short to perform regularly delete the policy will delete strategies and inert, like the case of the emergence of a waste of memory.

Therefore, if a regular basis in addition to the policy side, the server must be under the circumstances, a reasonable set of long-delete operation and execution frequency.

And in Redis is used to periodically delete and delete inert two strategies, well above the CPU and memory achieved a balance.

Inert delete

Inert expired key deletion policy implemented by db.c / expireIfNeeded function, Redis database of all read and write commands are invoked expireIfNeeded function before performing the Enter key to check:

• If the input key has expired, then expireIfNeeded function of the input key is deleted from the database.
• If the input key is not expired, then the function does expireIfNeeded action.

expireIfNeeded function like a filter, before it can actually execute the command to filter out expired input key, so as to avoid contact with the command key expires. Specific function code is as follows:

/*
* 检查 key 是否已经过期，如果是的话，将它从数据库中删除。
*
* 返回 0 表示键没有过期时间，或者键未过期。
*
* 返回 1 表示键已经因为过期而被删除了。
*/
int expireIfNeeded(redisDb *db, robj *key) {

    // 取出键的过期时间
    mstime_t when = getExpire(db,key);
    mstime_t now;

    // 没有过期时间
    if (when < 0) return 0; /* No expire for this key */

    // 如果服务器正在进行载入，那么不进行任何过期检查
    if (server.loading) return 0;

    /* If we are in the context of a Lua script, we claim that time is
    * blocked to when the Lua script started. This way a key can expire
    * only the first time it is accessed and not in the middle of the
    * script execution, making propagation to slaves / AOF consistent.
    * See issue #1525 on Github for more information. */
    now = server.lua_caller ? server.lua_time_start : mstime();

    /* If we are running in the context of a slave, return ASAP:
    * the slave key expiration is controlled by the master that will
    * send us synthesized DEL operations for expired keys.
    *
    * Still we try to return the right information to the caller, 
    * that is, 0 if we think the key should be still valid, 1 if
    * we think the key is expired at this time. */
    // 当服务器运行在 replication 模式时
    // 附属节点并不主动删除 key
    // 它只返回一个逻辑上正确的返回值
    // 真正的删除操作要等待主节点发来删除命令时才执行
    // 从而保证数据的同步
    if (server.masterhost != NULL) return now > when;

    // 运行到这里，表示键带有过期时间，并且服务器为主节点

    /* Return when this key has not expired */
    // 如果未过期，返回 0
    if (now <= when) return 0;

    /* Delete the key */
    server.stat_expiredkeys++;

    // 向 AOF 文件和附属节点传播过期信息
    propagateExpire(db,key);

    // 发送事件通知
    notifyKeyspaceEvent(REDIS_NOTIFY_EXPIRED,
        "expired",key,db->id);

    // 将过期键从数据库中删除
    return dbDelete(db,key);
}

Call expireIfNeeded command to delete expired key processes and execution get command is as follows (from "Design and Implementation of Redis second edition of" Chapter 9: Database):

Periodically delete

Expired keys periodically delete the policy implemented by the redis.c / activeExpireCycle function, call flow is serverCron () -> databasesCron () -> activeExpireCycle (). The core code is as follows (see for convenience of the core part, taken on the code):

int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
    ......

    // 对数据库执行各种操作
    databasesCron();

    ......
}

// 对数据库执行删除过期键，调整大小，以及主动和渐进式 rehash
void databasesCron(void) {

    // 函数先从数据库中删除过期键，然后再对数据库的大小进行修改

    /* Expire keys by random sampling. Not required for slaves
    * as master will synthesize DELs for us. */
    // 如果服务器不是从服务器，那么执行主动过期键清除
    if (server.active_expire_enabled && server.masterhost == NULL)
        // 清除模式为 CYCLE_SLOW ，这个模式会尽量多清除过期键
        activeExpireCycle(ACTIVE_EXPIRE_CYCLE_SLOW);

    /* Perform hash tables rehashing if needed, but only if there are no
    * other processes saving the DB on disk. Otherwise rehashing is bad
    * as will cause a lot of copy-on-write of memory pages. */
    // 在没有 BGSAVE 或者 BGREWRITEAOF 执行时，对哈希表进行 rehash
    if (server.rdb_child_pid == -1 && server.aof_child_pid == -1) {

        ......
        
    }
}
void activeExpireCycle(int type) {

    ......

    // 遍历数据库
    for (j = 0; j < dbs_per_call; j++) {
        int expired;
        // 指向要处理的数据库
        redisDb *db = server.db+(current_db % server.dbnum);

        // 为 DB 计数器加一，如果进入 do 循环之后因为超时而跳出
        // 那么下次会直接从下个 DB 开始处理
        current_db++;

        do {
            unsigned long num, slots;
            long long now, ttl_sum;
            int ttl_samples;

            // 获取数据库中带过期时间的键的数量
            // 如果该数量为 0 ，直接跳过这个数据库
            if ((num = dictSize(db->expires)) == 0) {
                db->avg_ttl = 0;
                break;
            }
            // 获取数据库中键值对的数量
            slots = dictSlots(db->expires);
            // 当前时间
            now = mstime();

            // 这个数据库的使用率低于 1% ，扫描起来太费力了（大部分都会 MISS）
            // 跳过，等待字典收缩程序运行
            if (num && slots > DICT_HT_INITIAL_SIZE &&
                (num*100/slots < 1)) break;

            /*  
            * 样本计数器
            */
            // 已处理过期键计数器
            expired = 0;
            // 键的总 TTL 计数器
            ttl_sum = 0;
            // 总共处理的键计数器
            ttl_samples = 0;

            // 每次最多只能检查 LOOKUPS_PER_LOOP 个键
            if (num > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP)
                num = ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP;

            // 开始遍历数据库
            while (num--) {
                dictEntry *de;
                long long ttl;

                // 从 expires 中随机取出一个带过期时间的键
                if ((de = dictGetRandomKey(db->expires)) == NULL) break;
                // 计算 TTL
                ttl = dictGetSignedIntegerVal(de)-now;
                // 如果键已经过期，那么删除它，并将 expired 计数器增一
                if (activeExpireCycleTryExpire(db,de,now)) expired++;
                if (ttl < 0) ttl = 0;
                // 累积键的 TTL
                ttl_sum += ttl;
                // 累积处理键的个数
                ttl_samples++;
            }

            ......

            // 已经超时了，返回
            if (timelimit_exit) return;

            // 如果已删除的过期键占当前总数据库带过期时间的键数量的 25 %
            // 那么不再遍历
        } while (expired > ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP/4);
    }
}

Some explanations:

1. serverCron () function is Redis timers, default run once every 100ms.
2. In databasesCron () function not only deletes expired keys were also carried out rehash operations.
3. If the server is not from the server, only the implementation of the initiative to clear expired.
4. ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP redis.h defined in 20. That delete key 20 expired. If you do not exceed the set time, each bank can delete multiple times.