Dictionary Dictionary Redis data structures

Dictionary (Dictionary), also known map (Map) or associative array (associative array), is an abstract data structure, a set of key-value pair (key-value pairs) composition, each key value pairs vary, program can add new key-value pairs to the dictionary, or based on key search, update, or delete files.

The dictionary has two main purposes:

Implement database key space (key space)
Hash keys as the underlying implementation of one type

Implement key database space

Redis is a key database, the database maintained by the key-value dictionary of: for each database has a corresponding dictionary that is called a space key (key space).

When a user adds a key-value pair to the database (regardless of what type of key-value pairs), the program added to the key-value pair key space; when a user deletes key-value pairs from the database, the program will be the key to be removed from the key space; and so on.

For example, the implementation of key FLUSHDB can empty space in all the key data:

redis> FLUSHDB
OK

Execution DBSIZE key space available key is returned to:

redis> DBSIZE
(integer) 0

SET also be provided with a key to a key space string, and the string value of the key taken from the key space with GET:

redis> SET number 10086
OK

redis> GET number
"10086"

redis> DBSIZE
(integer) 1

Hash keys as the underlying implementation of one type

Hash keys of Redis using the following two types as the underlying data structure implemented:

dictionary
Packing List

Because the compression dictionary lists more than save memory, so the program when creating a new Hash key, use the default list of compression as the underlying implementation, when necessary, the program will list the underlying transition from compression to the dictionary.

Hash when the user operates a key, the key at the bottom could be a hash table.

Dictionary of realization

Way to achieve the dictionary there are many:

List or array, high time complexity
Hash tables, both efficient and simple
Balanced tree, complex and stable and efficient

Among the many possible implementations, Redis select an efficient, simple hash table to achieve as the underlying dictionary.

/*
 * 字典
 *
 * 每个字典使用两个哈希表，用于实现渐进式 rehash
 */
typedef struct dict {

    // 特定于类型的处理函数
    dictType *type;

    // 类型处理函数的私有数据
    void *privdata;

    // 哈希表（2 个）
    dictht ht[2];

    // 记录 rehash 进度的标志，值为 -1 表示 rehash 未进行
    int rehashidx;

    // 当前正在运作的安全迭代器数量
    int iterators;

} dict;

Note that the use of two types of dict pointers, pointing to two hash tables.

Among them, 0 hash table (ht [0]) is the main use of hash table dictionary, and No. 1 hash table (ht [1]) is used only when the program numbers 0 to hash table rehash.

API：
Here Insert Picture Description

Hash table implementation

/*
 * 哈希表
 */
typedef struct dictht {

    // 哈希表节点指针数组（俗称桶，bucket）
    dictEntry **table;

    // 指针数组的大小
    unsigned long size;

    // 指针数组的长度掩码，用于计算索引值
    unsigned long sizemask;

    // 哈希表现有的节点数量
    unsigned long used;

} dictht;

attribute table is an array, each element of the array is a pointer pointing dictEntry structure.

Each dictEntry with a key-value pairs are stored, and a pointer to another structure dictEntry:

/*
 * 哈希表节点
 */
typedef struct dictEntry {

    // 键
    void *key;

    // 值
    union {
        void *val;
        uint64_t u64;
        int64_t s64;
    } v;

    // 链往后继节点
    struct dictEntry *next;

} dictEntry;

When a plurality of different keys have the same hash value,: dictEntry next attribute points to another structure, can be concatenated into a plurality dictEntry by the next pointer list, from where it can be seen, dictht processed using the method key chain address collision hash table with a list of these keys are connected to.

Hash Algorithm

Redis is currently using two different hash algorithms:

MurmurHash2 32 bit algorithm: the distribution rate and the speed of this algorithm are very good, specific information, please refer MurmurHash homepage: http://code.google.com/p/smhasher/ .
A case-insensitive hash algorithm djb algorithm based on: specific information, please refer http://www.cse.yorku.ca/~oz/hash.html .

Algorithm depends on the specific application data processed by the use of which:

Lua script command table and have used the algorithm 2 cache.
Algorithm 1 is broader: database, clustering, hash key, blocking operation and other functions are used in this algorithm.

rehash

For chain address method to solve the collision problem dictht hash table, the hash table performance depends on the ratio between the number (used attributes) size (size attribute) and save the node:

Size and number of nodes of the hash table, the ratio of 1: 1 performance, best hash table;
If the size of the number of nodes Bi Haxi table to be much larger, then the hash table will degenerate into a plurality of linked lists, hash table itself performance advantage would cease to exist;

In order to maintain good performance in case of increasing of the keys of the dictionary, the dictionary needs to be used for the hash table (ht [0]) rehash operations carried out: without any modification of the key, hash tables for expansion, as far as possible maintained at a ratio of about 1: 1.

dictAdd before each new key-value pair is added to the dictionary, will hash table ht [0] checks for ht [0] of size and used property, if the ratio therebetween ratio = used / size to meet any of the following a condition, then, rehash process will be activated:

NATURAL rehash: ratio> = 1, and the variable dict_can_resize is true.
Forced rehash: ratio greater than the variable dict_force_resize_ratio (current version, dict_force_resize_ratio value of 5).

When dict_can_resize be false?

When the application described earlier also spoke of the dictionary, the dictionary database is, the database hash key is the type of dictionary, when Redis (such as when executed BGSAVE or BGREWRITEAOF) when performing a background task persistence database using sub-process, in order to maximize of use of the system copy on write mechanism, the program will temporarily dict_can_resize set to false, avoid performing natural rehash, thereby reducing the program memory of touch (touch).

When the task is completed persistence, dict_can_resize will again be set to true.

On the other hand, when the dictionary satisfy the conditions for mandatory rehash, even dict_can_resize not true (or have BGSAVE BGREWRITEAOF being performed), the dictionary will be the same rehash.

Rehash execution

Dictionary rehash operations actually performs the following tasks:

Creating a ratio of ht [0] -> table greater ht [1] -> table, the size of at least ht [0] -> is used twice;
The ht [0] -> all key table to the migration ht [1] -> table, ht [0] -> table node will be gradually migrated to ht [1] -> table, as a fraction rehash performed several times (details explained in the next section), dictionary rehashidx variable will be recorded to rehash ht [0] of which index position. ;
The original ht [0] of the empty data and ht [1] is replaced with a new ht [0], create a new hash table is empty, and it is set ht [1], the attribute dictionary rehashidx is set to -1, identification rehash has stopped;

Progressive rehash

Progressive rehash mainly by _dictRehashStep and dictRehashMilliseconds two functions:

_dictRehashStep for the database dictionary, and the dictionary hash key passive rehash;
dictRehashMilliseconds routine tasks by Redis server program (server cron job) executed for the database dictionary for active rehash;

_dictRehashStep：

Each execution _dictRehashStep, [0] ht -> table of a hash table is not empty of all nodes on the index will be migrated to all ht [1] -> table.

After rehash start (d-> rehashidx not -1), each execution once to add, search, delete, _dictRehashStep will be executed once:

Because the dictionary will maintain the ratio of hash table size and the number of nodes within a small range, so the number of nodes on each index is not a lot (from the current version of rehash conditions, the average only one, at most not usually will be more than five), so while performing operations on a single index node migration, almost no impact on response time.

dictRehashMilliseconds：

dictRehashMilliseconds may be within a specified number of milliseconds, the dictionary for rehash.

When the Redis server to perform routine tasks, dictRehashMilliseconds will be executed within a specified period of time, as much as possible for the dictionary database dictionary that need to be rehash rehash, rehash the dictionary database to accelerate the process of (progress).

Other measures:

When the hash table rehash, the dictionary will take special measures to ensure the smooth rehash, correctly:

Because when rehash, the dictionary will use two hash tables, so all looks in this period, delete, etc., in addition to [0] on ht, also need to be in [1] on ht.
When the add operation, a new node is added directly to ht [1] instead of ht [0], thus ensuring ht [0] is the number of nodes in the whole process only rehash Save does not increase.

Dictionary of contraction

Above rehash of the chapter describes the situation be extended dictionary by rehash (expand), if available nodes of the hash table is larger than the number of nodes with many words, it is also possible to shrink (shrink by a hash table rehash )dictionary.

Rehash rehash expansion and contraction exhibited almost the same as the above operation, the following steps:

Create a [0] ratio ht -> table small ht [1] -> table;
The ht [0] -> all key table to the migration ht [1] -> table;
The ht [0] of the empty original data, and [1] is replaced with a new ht [0] ht;

Dictionary contraction rules are defined by redis.c / htNeedsResize function:

/*
 * 检查字典的使用率是否低于系统允许的最小比率
 *
 * 是的话返回 1 ，否则返回 0 。
 */
int htNeedsResize(dict *dict) {
    long long size, used;

    // 哈希表大小
    size = dictSlots(dict);

    // 哈希表已用节点数量
    used = dictSize(dict);

    // 当哈希表的大小大于 DICT_HT_INITIAL_SIZE
    // 并且字典的填充率低于 REDIS_HT_MINFILL 时
    // 返回 1
    return (size && used && size > DICT_HT_INITIAL_SIZE &&
            (used*100/size < REDIS_HT_MINFILL));
}

By default, REDIS_HT_MINFILL is 10, That is, when the filling rate is less than 10% of the dictionary, the program can be operated to shrink the dictionary.

One difference dictionary contraction and expansion of the dictionary is: dictionary extended operation is automatically triggered (either automatically extended or mandatory extension); and the dictionary shrink operation is performed by the program manually.

Therefore, the use of the dictionary program can decide when to shrink the dictionary:

When the dictionary used to implement the hash key, and every time a key is deleted from the dictionary of the program will be executed once htNeedsResize function, if the contraction reached the standard dictionary, the program will immediately dictionary contraction;
When the dictionary used to implement the database key space (key space), the contraction of the timing of the decision by the redis.c / tryResizeHashTables function

Dictionary iteration

With its own dictionary iterators implement - the dictionary to iterate on the hash table is actually used to iterate the dictionary:

Iterator first iteration of the first dictionary of the hash table, and then, if rehash progress then it continues for a second iteration of the hash table.
When the iterative hash table, find the first index is not empty, then iterate all nodes on this index.
When the index iteration is over, the next we continue to find an empty index, so repeatedly, until the entire hash table are iterative last.

Dictionary iterators in two ways:

Security iterator: In the iteration process, you can modify the dictionary.
Unsafe iterator: In the iteration process, the right to modify the dictionary.

The following data structure is defined iterator:

/*
 * 字典迭代器
 */
typedef struct dictIterator {

    dict *d;                // 正在迭代的字典

    int table,              // 正在迭代的哈希表的号码（0 或者 1）
        index,              // 正在迭代的哈希表数组的索引
        safe;               // 是否安全？

    dictEntry *entry,       // 当前哈希节点
              *nextEntry;   // 当前哈希节点的后继节点

} dictIterator;