[redis] 5 data structures of redis and their underlying implementation principles

Data structure in redis

Redis supports five data types: string (string), hash (hash), list (list), set (unordered collection) and zset (ordered collection).

In the seckill project, I used the Set and Hash structures of redis:

• String: A key corresponds to a string, and string is the most basic data type of Redis. (The byte abase framework only implements the string data structure of redis, so if we want to store complex data structures, we can only convert them into strings in json format for storage)
• list: A key corresponds to a list of strings. The bottom layer is implemented by a doubly linked list, and it supports many operations supported by a doubly linked list.
• Hash：
• Set: For example, an instance of Set: A = {'a', 'b', 'c'}, A is the key of the set, 'a', 'b' and 'c' are the members of the set. Unordered, no repeating elements.
• SortedSet: A score is added to the set, and the data in the set is ordered.

The underlying implementation of redis data structure

string

Use a 简单动态字符串（SDS）的数据类型call to achieve.

/*  
 * 保存字符串对象的结构  
 */  
struct sdshdr {
    
      
    int len;  // buf 中已占用空间的长度  
    int free;  // buf 中剩余可用空间的长度
    char buf[];  // 数据空间  
};

Advantages of SDS over C strings:

• SDS saves the length of the string, but the C string does not save the length. It is necessary to traverse the entire array (until '\0' is found) to get the length of the string.
• When modifying SDS, check whether the given SDS space is sufficient, if not, expand the SDS space first to prevent buffer overflow. The C string does not check whether the string space is sufficient, and it is easy to cause a buffer overflow when calling some functions (such as the strcat string concatenation function).
• The mechanism of SDS pre-allocating space can reduce the number of times to reallocate space for strings.

list

使用双向链表来实现。

hash

The hash structure is actually a dictionary with many key-value pairs (similar to python's dict type).

Redis's hash table is a dictht structure:

typedef struct dictht {
    
    
   dictEntry **table;//哈希表数组   
   unsigned long size;//哈希表大小  
   unsigned long sizemask;//哈希表大小掩码，用于计算索引值
   unsigned long used;//该哈希表已有节点的数量
}

The structure of the hash table node is as follows:

typeof struct dictEntry{
    
      
   void *key;//键
   union{
    
      //不同键对应的值的类型可能不同，使用union来处理这个问题
      void *val;
      uint64_tu64;
      int64_ts64;
   }
   struct dictEntry *next;
}

One of the methods to resolve hash conflicts is the zipper method.

In order to keep the loading factor of the hash table within a reasonable range, it is necessary to expand or shrink the size of the hash table, which is called rehash. There are a total of two hash table dictht structures in the dictionary, ht[0] is used to store key-value pairs, ht[1] is used to temporarily store data during rehash, usually the hash table it points to is empty and needs to be expanded or contracted ht[0]'s hash table is allocated space for it.

For example, expanding the hash table is to allocate a space twice the size of ht[0] for ht[1], then migrate all the data of ht[0] to ht[1] through rehash, and finally release ht[0] ], make ht[1] become ht[0], and assign an empty hash table to ht[1]. Shrinking hashtables is similar.

Progressive rehash: redis does not specifically find time to perform rehash at one time, but gradually. During rehash, it does not affect external access to ht[0]. It is required to synchronize the corresponding data to ht[1] when modifying the dictionary , when all data transfer is completed, rehash ends.

set

set can be implemented with intset or dictionary.

intset

Only when the data is all integer values ​​and the number is less than 512, use intset. Intset is an ordered set composed of integers, which can be used for binary search.

dictionary

The dictionary (zipper method) is used when the conditions for using intset are not met, and the value is set to null when using the dictionary.

check

Each element in zset contains the data itself and a corresponding score (score).

Classic example: the key of a zset is "math", which represents the grades of the math class, and then a lot of data can be inserted into this key. When entering data, each time you need to enter a name and a corresponding score. Then the name is the data itself, and the score is its score.

The data of zset itself does not allow duplication, but the score allows duplication.

The underlying implementation principle of zset:

1. When the data is small, use ziplist: ziplist occupies continuous memory, and each element is stored continuously in the form of (data + score), sorted by score from small to large. In order to save memory, the space occupied by each element of ziplist can be different. For large data (long long), more bytes are used for storage, and for small data (short), less bytes are used for storage. Therefore, it is necessary to traverse in order when searching. Ziplist saves memory but has low search efficiency.
2. When there is a lot of data, use a dictionary + skip table:
   

The dictionary is used to look up the score based on the data, and the jump table is used to look up the data based on the score (high search efficiency).

Theoretically speaking, red-black trees can also complete the operations of search, insertion, deletion, and iterative output of ordered sequences, and the time complexity is the same as that of jump tables.

The reason why redis uses skip table instead of red-black tree:

1. In the operation of searching data according to the interval, the efficiency of the red-black tree is not as high as that of the jump table. The jump table can locate the starting point of the interval in O(logn) time complexity, and then query backwards sequentially in the original linked list.
2. Compared with the red-black tree, the jump table also has the advantages of easier code implementation, good readability, less error-prone, and more flexible.
3. When inserting or deleting, only a few nodes need to be adjusted in the jump table, while the red-black tree needs to be repainted and rotated, which is expensive.

Skip table insertion and deletion process

The jump table is constructed based on an ordered single-linked list. The search efficiency is improved by building an index, and space is exchanged for time. The search method is to search from the top linked list layer to the bottom, and finally find the corresponding node in the bottom linked list:

• Insert: Find the position layer by layer, and then insert into the bottom linked list. Note that it is necessary to maintain the size balance between the index and the original linked list. If the number of underlying nodes increases, the index will also increase accordingly, so as to avoid too many nodes between the two indexes and reduce the search efficiency. Similarly, when the underlying nodes are greatly reduced, the index is also reduced accordingly.

• Delete: If this node also appears in the index, then in addition to deleting the node in the original linked list, the node in the index must also be deleted.

  The time complexity of skip table lookup is O(log(n)). The space complexity occupied by the index is O(n).

• Time complexity: Time complexity = the number of layers of the index * the number of elements traversed by each layer of index.

  First look at the number of index layers. Assuming that every two nodes are selected as nodes of the upper-level index, and the highest-level index has 3 nodes, the number of index layers is log2(n).

  Then look at how many elements are traversed in each layer. First, the highest layer can traverse up to 3 nodes, and then it can go down. Similarly, the second-level layer can also traverse up to three nodes, and then it can go down. After taking the average, it can be considered that each layer traverses 2 nodes.

  Therefore, time complexity = 2log2(n). Similarly, if an index is taken for every k nodes, it is klogk(n)

• Space complexity: Take an index for every two nodes as an example. There are n nodes in the first layer, n/2 in the second layer, and n/4 in the third time. The sum of proportional sequences, or the limit, can be considered as an index The number of nodes is infinitely close to n, so the space complexity is O(n).