Refine the hash object of Redis high-performance data structure

background

The last section talked about Redis's high-performance string structure SDS , today we look at the hash object of redis.

Hash object

Introduction

• There are two encoding (underlying implementation) methods for redis hash objects, dictionary encoding and compressed list encoding. When using dictionary encoding, the program saves the key of the hash table as the key of the dictionary, and the value of the hash as the value of the dictionary. The key value of the dictionary is of string type.
• The length of the key and value strings of all key-value pairs stored in the hash object is less than 64 bytes and the number of key-value pairs stored in the hash object is less than 512. The ziplist is used, and the hashtable ( Dictionary encoding)

Deep understanding

ZipList (compressed list)

1. The compressed list of redis is a continuous memory space, and the elements are stored next to each other, without any redundant space
2. Source code

struct ziplist<T> {
int32 zlbytes; // 整个压缩列表占用字节数
int32 zltail_offset; // 最后一个元素距离压缩列表起始位置的偏移量，用于快速定位到最后一个
节点
int16 zllength; // 元素个数
T[] entries; // 元素内容列表，挨个挨个紧凑存储
int8 zlend; // 标志压缩列表的结束，值恒为 0xFF
}


/**
*entry对象源码
*/
struct entry {
int<var> prevlen; // 前一个 entry 的字节长度
int<var> encoding; // 元素类型编码
optional byte[] content; // 元素内容
}

3. The compressed list supports two-way traversal, so there is the field zltail_offset, which can quickly locate the last element. Then search in reverse order (O(1))
4. Prevlen means the length of the previous field. Some people have questions. Why is the length of the previous entry? Why is it not your own? In fact, it has another role to use when compressing the list flashback traversal. Quickly locate the location of the next element. Because it is a continuous storage space, you can determine the location of an entry by knowing the location of the current element + this space address. Why is this so? Because the size of the entry is different. If they are the same, they can behave according to the following table (personal understanding, please point out if there are errors), and prevlen is a variable-length integer. The normal operation of redis uses different data types for different lengths. Save memory
5. Encoding means the encoding type of the element. With this field, the setting of the element content and the allocation of memory size can be determined. A way to prevent waste of memory allocation. See the specific content below

1、00xxxxxx 最大长度位 63 的短字符串，后面的 6 个位存储字符串的位数，剩余的字
节就是字符串的内容。
2、01xxxxxx xxxxxxxx 中等长度的字符串，后面 14 个位来表示字符串的长度，剩余的
字节就是字符串的内容。
3、10000000 aaaaaaaa bbbbbbbb cccccccc dddddddd 特大字符串，需要使用额外 4 个字节
来表示长度。第一个字节前缀是 10，剩余 6 位没有使用，统一置为零。后面跟着字符串内
容。不过这样的大字符串是没有机会使用的，压缩列表通常只是用来存储小数据的。
4、11000000 表示 int16，后跟两个字节表示整数。
5、11010000 表示 int32，后跟四个字节表示整数。
6、11100000 表示 int64，后跟八个字节表示整数。
7、11110000 表示 int24，后跟三个字节表示整数。
8、11111110 表示 int8，后跟一个字节表示整数。
9、11111111 表示 ziplist 的结束，也就是 zlend 的值 0xFF。
10、1111xxxx 表示极小整数，xxxx 的范围只能是 (0001~1101), 也就是 1~13，因为
0000、1110、1111 都被占用了。读取到的 value 需要将 xxxx 减 1，也就是整数 0~12 就是
最终的 value。

6. There are two prerequisites for selecting a compressed list for hash objects. One of them is that the size of the key value is less than 64. Why is less than 64 and the key value pair is less than 512. I won’t say it specifically. You can combine SDS. Think about the expansion method in, there is no redundant space in the compressed list. Frequent expansion will occur during expansion. In addition, copying data after taking up a large space is a waste of performance. So when the amount of data is large, another data structure is chosen, which is hashtable (dictionary)

HashTable (dictionary)

Introduction

• The implementation of hashtable in redis and hashMap in java is similar, and both are implemented through arrays and linked lists. That is, the key-value form. Of course, it also uses the chain address method to resolve hash conflicts (you can think of several ways to resolve hash conflicts). When different keys create the same hash value, vlue is placed on the linked list, as shown below.

• There is still a big difference between the details and the hashMap in java. Examples include the expansion process, the key value hash algorithm, and so on. Next, we will make a detailed product based on the source code.
  

• The official explanation: a dictionary, also known as a map or an associative array, is an abstract data structure composed of a set of key-value pairs, and each key-value pair The keys are different, the program can add new key-value pairs to the dictionary, or search, update or delete based on the keys

The underlying structure of the dictionary uses the dict in redis. Not only is the dict used at the bottom of the hash object, but also the key-vlue structure is used globally in redis, which is the form of a dictionary, and the bottom layer of the data structure of Zset is also based on the dict structure in redis. Let's take a look at its source code:

// resdis 全局使用的字典结构
struct RedisDb {
dict* dict; // all keys key=>value
dict* expires; // all expired keys key=>long(timestamp)
...}

// 有序集合的底层数据结构
struct zset {
dict *dict; // all values value=>score
zskiplist *zsl;
}

2. Deep analysis of dict structure

• Source code:

/*
 * 字典
 *
 * 每个字典使用两个哈希表，用于实现渐进式 rehash
 */
typedef struct dict {

    // 特定于类型的处理函数
    dictType *type;

    // 类型处理函数的私有数据
    void *privdata;

    // 哈希表（2 个）
    dictht ht[2];

    // 记录 rehash 进度的标志，值为 -1 表示 rehash 未进行
    int rehashidx;

    // 当前正在运作的安全迭代器数量
    int iterators;

} dict;

• You can see that there are two hashtables in the member variables of the class, usually one has a value and the other has no value. The problem we encountered in the compressed list is the performance problem in expansion. These two hashtables are to solve the expansion problem. Gradually relocate during expansion and contraction. When the relocation ends, the old hashtable will be deleted and the new hashtable will be replaced.
• Then let's take a closer look at hashtable, (hashtable in Java is a thread-safe version of hashMap in Java). The hashtable here is similar to the hashmap in java, and the way to resolve hash conflicts is through bucketing. One-dimensional array, two-dimensional linked list. But there are still some differences in expansion.

struct dictEntry {
void* key;
void* val;
dictEntry* next; // 链接下一个 entry
}
struct dictht {
dictEntry** table; // 二维
long size; // 第一维数组的长度
long used; // hash 表中的元素个数
...
}

• Let's take a look at how the hash is performed in redis.
  1. The expansion of the large dictionary is very time-consuming. You need to reapply for a new array, and then reattach all the elements in the linked list of the old dictionary to the new array. This The process time complexity is O(n). As a single-threaded redis, how can redis waste time here? So he adopted the progressive processing method (when it comes to the progressive method, can you think of his progressive batch query based on the key scan and keys), click here for the rehash process . The idea is the small step execution we mentioned above.
• The Set structure is also implemented through a dictionary, but not all values ​​are NULL, have you thought of anything? Is the hashSet in Java similar to this? .

to sum up

1. There are two underlying implementations of hash objects, hashtable (dictionary) and ziplist (compressed linked list)
2. Because the compressed linked list is a continuous space, the performance is significant when the amount of data is small at the beginning, but the problem of slow expansion will occur when the amount of data is large.
3. The dictionary solves the problem of the expansion of the compressed list through the double-hahstable method and the progressive hash method.
4. Redis high-performance data structure, we can see that he has a lot of details about the details, such as different number sizes use different field types, and the same object selects different storage types according to the size. (Save memory)