Redis (3) storage principle and data model (hash conflict, progressive rehash)

Redis series of articles

Redis (1) principle and basic commands (flexible array)
Redis (2) network protocol and asynchronous mode (optimistic lock & pessimistic lock)
Redis (3) storage principle and data model (hash conflict, progressive rehash)
Redis jump table

---

1. Redis storage structure

Redis is a key-value structure, where value includes: dictionary, doubly linked list, compressed list, skip list, integer array, dynamic string.

storage conversion

The data structure of each value in redis has different automatic storage conversions according to different situations.

Key-value store implementation

The KV organization in redis is implemented through a dictionary, that is, a hash table. The key string is calculated by the hash function to obtain a 64-bit integer;

hash conflict

If redis uses hash tables to store key-values, it will encounter hash conflicts. Common hash conflict resolution methods are:

• Open chain method: connect the value values ​​of hash conflicts with a linked list, and a linked list is connected under the key value of each hash result. If the linked list is too long, the linked list can be converted into a red-black tree for optimization.
• rehash: Increase the size of the hash bucket. Redis has two hash tables. When hash table 1 conflicts, hash table 2 will be used, and the hash bucket size of hash table 2 is usually twice that of hash table 1.

However, when rehash, copying the data in hash table 1 to hash table 2 is a huge project, which may cause redis thread blocking and affect redis performance. Therefore, redis has a progressive rehash mechanism to solve this problem.

Progressive rehash

Progressive rehash, like rehash, also needs to copy the data in hash table 1 to hash table 2, but this copy process is not a one-off, but a step-by-step block transfer.

The progressive of redis has two rules:

1. The idea of ​​​​divide and conquer, divide rehash into each step of adding, deleting, modifying and checking operations, that is, copy a key every time redis is processed;
2. In the timer, a maximum of one millisecond rehash is executed; 100 array key slots are copied each time;

What will happen if there are additions, deletions, checks and changes during the rehash process?

• Search: When searching for data, it will first search for data in hash table 1, and if it is not found, it will search for data in hash table 2.
• Add: When new data is added, it will only be added to hash table 2, and no operation will be performed on hash table 1.
• Delete & modify: can operate on both tables.

In addition to rehash expansion, redis will also use rehash when shrinking capacity.
But in the rehash phase, expansion and contraction will no longer occur. Must wait for rehash to end.

Big KEY

In the redis instance, if a large object is formed, such as a large hash or a large zset, when such an object is expanded, it will apply for a larger piece of memory at one time, which will cause a freeze; if this If the large key is deleted, the memory will be recycled at one time, and the freeze phenomenon will occur again; if the memory of redis is observed to fluctuate greatly, it is most likely caused by the large key;

Solution:

• split value
• compressed value
• Delete non-hotspot large keys (can be deleted using redis asynchronous mechanism)

jump table

The sorted set (Sorted Set) in Redis is implemented with a skip list (Skip list). The jump table is to solve the problem of high query time complexity of the linked list in the ordered node scenario. The time complexity can reach O(logn).
For details, refer to Redis jump table