See redis source data structure from (iii) a hash table

See redis source data structure from (iii) a hash table

I humble junior, in preparation for next year's spring strokes, below is a review of some of their notes, there may be wrong places, but also invited the community chiefs timely advice

Dictionary relative to the arrays, linked lists, it is a higher-level data structures, like our Chinese dictionary, like, you can uniquely identify a Chinese character by pinyin or radical, in a program where we each tube called a mapping between a key-value pair a lot of key-value pairs together constitute our dictionary structure.

There are many advanced dictionary structure to achieve, for example, our underlying HashMap in Java implementation, according to Hash values ​​of key value pairs of uniform will disperse to the array, and in the face of hash collision, key violation of the one-way chain series, and more than eight nodes in the linked list structure fission into red-black tree.

So redis is how to achieve it? We take a look.

A, redis underlying hash dictionary definition

hash dictionary redis is to use a hash table zipper law

/*
 * 哈希表节点
 */
typedef struct dictEntry {

    // 键
    void *key;

    // 值
    union {
        void *val;
        uint64_t u64;
        int64_t s64;
    } v;

    // 链往后继节点（拉链法）
    struct dictEntry *next; 

} dictEntry;

Its essence is in the form of a single list plus an array of arrays used to store key, when faced with a conflict of different hash key is mapped to the same array position, take the zipper method put in the same array position to resolve the conflict, it is similar to the hashmap

/*
 * 哈希表
 */
typedef struct dictht {

    // 哈希表节点指针数组（俗称桶，bucket）
    dictEntry **table;      

    // 指针数组的大小
    unsigned long size;     

    // 指针数组的长度掩码，用于计算索引值
    unsigned long sizemask; 

    // 哈希表现有的节点数量
    unsigned long used;     

} dictht;

The above is the realization of the underlying dictionaries ---- hash table, hash table is actually two

/*
 * 字典
 *
 * 每个字典使用两个哈希表，用于实现渐进式 rehash
 */
typedef struct dict {

    // 特定于类型的处理函数
    dictType *type;

    // 类型处理函数的私有数据
    void *privdata;

    // 哈希表（2个）
    dictht ht[2];       

    // 记录 rehash 进度的标志，值为-1 表示 rehash 未进行
    int rehashidx;

    // 当前正在运作的安全迭代器数量
    int iterators;      

} dict;

Using two hash tables implementing the dictionary is a dictionary for a subsequent use of the extended rehash

• ht [0]: is usually used to store the common key of the hash table, and a dictionary in this
• ht [2]: is enabled only when performing rehash dictionary expansion

Second, a dictionary-related operations

1. Initialization dictionary

In redis in the dictionary hash table is initialized using delay tactics and did not allocate memory for the hash table only when the data is first inserted, really allocate memory when creating the dictionary. Look at the dictionary to create a function dictCreate.

/*
 * 创建一个新字典
 *
 * T = O(1)
 */
dict *dictCreate(dictType *type,
        void *privDataPtr)
{
    // 分配空间
    dict *d = zmalloc(sizeof(*d));

    // 初始化字典
    _dictInit(d,type,privDataPtr);

    return d;
}

/*
 * 初始化字典
 *
 * T = O(1)
 */
int _dictInit(dict *d, dictType *type,
        void *privDataPtr)
{
    // 初始化 ht[0]
    _dictReset(&d->ht[0]);

    // 初始化 ht[1]
    _dictReset(&d->ht[1]);

    // 初始化字典属性
    d->type = type;
    d->privdata = privDataPtr;
    d->rehashidx = -1;
    d->iterators = 0;

    return DICT_OK;
}

2.rehash

Just mentioned dictionary ht at the top [1] is enabled only when the rehash, rehash then what is it?

rehash is when the hash table key expansion or readjust respective hash value after the volume reduction

rehash process:

1. Progressive initialization rehash

Progressive rehash initialization actually just created a new hash table, and not the original hash table of key rehash and carried into the new table

/* Expand or create the hash table */
/*
 *  渐进式rehash初始化
 *  创建一个新哈希表，并视情况，进行以下动作之一：
 *  
 *   1) 如果字典里的 ht[0] 为空，将新哈希表赋值给它
 *   2) 如果字典里的 ht[0] 不为空，那么将新哈希表赋值给 ht[1] ，并打开 rehash 标识
 *
 * T = O(N)
 */
int dictExpand(dict *d, unsigned long size)
{
    dictht n; /* the new hash table */
    
    // 计算哈希表的真实大小
    // O(N)
    unsigned long realsize = _dictNextPower(size);

    /* the size is invalid if it is smaller than the number of
     * elements already inside the hash table */
    //如果当前正在进行rehash或者新哈希表的大小小于现已使用
    if (dictIsRehashing(d) || d->ht[0].used > size)
        return DICT_ERR;

    /* Allocate the new hash table and initialize all pointers to NULL */
    // 创建并初始化新哈希表
    // O(N)
    n.size = realsize;
    n.sizemask = realsize-1;
    n.table = zcalloc(realsize*sizeof(dictEntry*));
    n.used = 0;

    /* Is this the first initialization? If so it's not really a rehashing
     * we just set the first hash table so that it can accept keys. */
    // 如果 ht[0] 为空，那么这就是一次创建新哈希表行为
    // 将新哈希表设置为 ht[0] ，然后返回
    if (d->ht[0].table == NULL) {
        d->ht[0] = n;
        return DICT_OK;
    }

    /* Prepare a second hash table for incremental rehashing */
    // 如果 ht[0] 不为空，那么这就是一次扩展字典的行为
    // 将新哈希表设置为 ht[1] ，并打开 rehash 标识
    d->ht[1] = n;
    d->rehashidx = 0;

    return DICT_OK;
}

2. Insert the key into a rehash of a new hash table

In redis achieved, there is no concentration of the original key to re rehash new slot, but the decomposition to execute each command, and a periodic function, such as change search function will be determined in each of the deletions redis dictionary hash table is in progress progressive rehash, namely (rehashidx! = -1), and if so, to help perform a rehash on the key after the new hash table in the process

Add key-value pairs such as operating below the

    // 判断是否在进行渐进式的初始化rehash，如果是，则尝试渐进式地 rehash 一个元素
    if (dictIsRehashing(d)) _dictRehashStep(d);

Details see below:

Insert key-value pairs

/*
 * 添加给定 key-value 对到字典
 *
 * T = O(1)
 */
int dictAdd(dict *d, void *key, void *val)
{
    // 添加 key 到哈希表，返回包含该 key 的节点
    dictEntry *entry = dictAddRaw(d,key);

    // 添加失败？
    if (!entry) return DICT_ERR;

    // 设置节点的值
    dictSetVal(d, entry, val);
	
    //添加成功
    return DICT_OK;
}

The actual add function:

/*
 * 添加 key 到字典的底层实现，完成之后返回新节点。
 *
 * 如果 key 已经存在，返回 NULL 。
 *
 * T = O(1)
 */
dictEntry *dictAddRaw(dict *d, void *key)
{
    int index;
    dictEntry *entry;
    dictht *ht;

    // 判断是否在进行渐进式的初始化rehash，如果是，则尝试渐进式地 rehash 一个元素
    if (dictIsRehashing(d)) _dictRehashStep(d);

    // 查找可容纳新元素的索引位置
    // 如果元素已存在， index 为 -1
    if ((index = _dictKeyIndex(d, key)) == -1)
        return NULL;//key存在直接返回null

    /* Allocate the memory and store the new entry */
    // 决定该把新元素放在哪个哈希表，如果实在正在进行渐进式rehash初始化，则新元素放在ht[1]，否则则放在ht[0]
    ht = dictIsRehashing(d) ? &d->ht[1] : &d->ht[0];
    // 为新元素分配节点空间
    entry = zmalloc(sizeof(*entry));
    // 新节点的后继指针指向旧的表头节点
    entry->next = ht->table[index];
    // 设置新节点为表头
    ht->table[index] = entry;
    // 更新已有节点数量
    ht->used++;

    /* Set the hash entry fields. */
    // 关联起节点和 key
    dictSetKey(d, entry, key);

    // 返回新节点
    return entry;
}

rehash process

In fact, adding the key top is performed by calling the execution rehash dictRehash method, a function formula rehash progressive steps performed N, n is the number of rehash

/*
 * 执行 N 步渐进式 rehash 。
 *
 * 如果执行之后哈希表还有元素需要 rehash ，那么返回 1 。
 * 如果哈希表里面所有元素已经迁移完毕，那么返回 0 。
 *
 * 每步 rehash 都会移动哈希表数组内某个索引上的整个链表节点，
 * 所以从 ht[0] 迁移到 ht[1] 的 key 可能不止一个。
 *
 * T = O(N)
 */
int dictRehash(dict *d, int n) {
    //判断当前有没有在进行rehash初始化。没有则返回0
    if (!dictIsRehashing(d)) return 0;

    while(n--) {
        dictEntry *de, *nextde;

        // 如果 ht[0] 已经为空，那么迁移完毕
        // 用 ht[1] 代替原来的 ht[0]
        if (d->ht[0].used == 0) {

            // 释放 ht[0] 的哈希表数组
            zfree(d->ht[0].table);

            // 将 ht[0] 指向 ht[1]
            d->ht[0] = d->ht[1];

            // 清空 ht[1] 的指针
            _dictReset(&d->ht[1]);

            // 关闭 rehash 标识
            d->rehashidx = -1;

            // 通知调用者， rehash 完毕
            return 0;
        }

        /* Note that rehashidx can't overflow as we are sure there are more
         * elements because ht[0].used != 0 */
        assert(d->ht[0].size > (unsigned)d->rehashidx);
        // 从当前正在rehash初始化的索引开始，移动到数组中首个不为 NULL 链表的索引上
        while(d->ht[0].table[d->rehashidx] == NULL) d->rehashidx++;
        // 指向链表头
        de = d->ht[0].table[d->rehashidx];
        // 将链表内的所有元素从 ht[0] 迁移到 ht[1]
        // 因为桶内的元素通常只有一个，或者不多于某个特定比率
        // 所以可以将这个操作看作 O(1)
        while(de) {
            unsigned int h;

            nextde = de->next;

            /* Get the index in the new hash table */
            // 计算元素在 ht[1] 的哈希值
            h = dictHashKey(d, de->key) & d->ht[1].sizemask;

            // 添加节点到 ht[1] ，调整指针
            de->next = d->ht[1].table[h];
            d->ht[1].table[h] = de;

            // 更新计数器
            d->ht[0].used--;
            d->ht[1].used++;

            de = nextde;
        }

        // 设置指针为 NULL ，方便下次 rehash 时跳过
        d->ht[0].table[d->rehashidx] = NULL;

        // 前进至下一索引
        d->rehashidx++;
    }

    // 通知调用者，还有元素等待 rehash
    return 1;
}

After the execution, the original ht [0] list node .table [rehashidx] at all transferred to the ht [1]

rehash Summary

In redis, the need to expand or contract all of the key hash table ht [0] to the inside of rehash ht [1] which, however, the operation is not disposable rehash centralized done, but division multiple times, progressively completed. Rehash order to avoid impact on the performance of the server, the server will not disposable all keys ht [0] to all the inside to rehash ht [1], but multiple times, to progressively ht [0] key inside slowly to rehash of ht [1].

The following is a detailed step of rehash incremental hash table:

• As ht [1] allocate space for the dictionary also holds ht [0] and ht [1] two hash tables.

• Maintain an index counter variable rehashidx in the dictionary and set its value to 0, it represents rehash the work began.

• During rehash carried out every time the dictionary add, delete, search or update operation, in addition to a program other than the specified action, but also incidentally the ht [0] hash table on rehashidx index of all key-value pairs to rehash ht [1], when rehash completed, the program will increase a value of the attribute rehashidx.

• With the operation of the dictionary performed, eventually a point in time, ht [0] to all key-value pairs will be to rehash ht [1], In this case the program will rehashidx attribute value to -1, indicating that the operation rehash completed.

The benefits of progressive rehash that it take the form of divide and rule will rehash key beach to add to the dictionary for each of the required calculations are, delete, search and update operations on avoiding centralized rehash brought a large amount of calculation.