Article directory
Dynamic String SDS
We all know that the keys we save in Reids are all in the form of strings, and the value is often a string or a collection of strings. Visible string is also the most commonly used data structure in Redis.
We also know that the bottom layer of Redis is written in C language, but Redis does not use C language strings directly in strings, because for Redis, C language strings will have many problems. does not fully meet the needs.
Because in the C language, the bottom layer of the string is a character array, so if you want to get the length of the string, you must go through a loop operation to get the result, and get the length of the string directly without sending it ;
And in the C language, the end of its string will use '\0' to mark the end of the language, which is non ;
In the C language, once the string is created, it is stored in the constant pool, and stored in the constant pool, we all know that it cannot be modified .
And it is for these reasons that Redis created a string structure that conforms to itself, calledSimple Dynamic String (Simple Dynamic String), referred to as SDS.
Let's first look at the underlying source code when creating SDS: (written in C language)
struct __attribute__ ((__packed__)) sdshdr8 {
/* uint8_t 中的8表示占 8 个字节 */
uint8_t len; /* 表示已保存的字符串的字节数，不包含表示结尾的标识符 */
uint8_t alloc; /* 表示申请的字节数，不包含表示结尾的标识符 */
unsigned char flags; /* 表示不同 SDS 的头部类型，用来控制 SDS 的头大小 */
char buf[]; /* 保存存进来的字符串 */
};
Among them, we can clearly see that the maximum number of stored string bytes is only 2 to the 8th power (-257 ~ +255), which is obviously very unreasonable, but don't panic, Redis will definitely not make this mistake It is a very low-level operation. In the specific implementation of Redis, it implements a concept similar to polymorphism in Java. (5 in total) The flags here are a kind of control for this version.
This obviously meets our needs.
We keep saying that SDS is a dynamic string.
For example: we now want to create a string "hi" of the SDS.
Then the structure it initially creates is:
Now we want to append a string ",Tom" to SDS, at this timeRedis will first apply for a new memory space:
If the new string is less than 1M, the new space is twice the length of the extended string + 1;
If the new character string is larger than 1M, the new space is the length of the character string after expansion+1M+1.
This method of allocating a memory space that is slightly larger than the actual needs is called memory pre-allocation .
Then the structure of the current SDS is as follows:
As for Shangmin’s expansion mechanism, 1 must be added at the end to save the identifier at the end of the string, but in SDS, the symbol that marks the end of the string is the length of the previous len, so in essence, it can save Envoy identifier, but its use is not recommended.
Advantages of SDS
- The time complexity of getting the length of a string is O(1)
- Support dynamic expansion
- Reduce the number of memory allocations (dynamic)
- It achieves binary security (because in Redis, the length of the read string is determined by the header information)
IntSet
IntSet is an implementation of the Set collection in Redis. It is implemented based on an integer array and has the characteristics of being readable, variable, ordered, and unique.
First look at its source code:
typedef struct intset {
uint32_t encoding; /* 表示编码方式，支持存放16位、32位、64位 */
uint32_t length; /* 表示元素的个数 */
int8_t contents[]; /* 整数的数组，保存集合数据 */
} intset;
Among them, the encoding consists of three modes, which respectively represent the size of the stored integer.
In IntSet, in order to facilitate searching, Redsi will store all the data in IntSet in the contents array in ascending order.
Now we want to save an array with an array of 5, 10, and 20, so its underlying implementation is:
Now, each number in the array is within the range of int16_t, so the encoding method adopted is INTSET_ENC_INT16, and the byte size occupied by each part is:
encoding: 4 bytes
length: 4 bytes
contents: 2 bytes * 3 = 6 byte
At this time, we want to add a number to it: 5000, but this number is now beyond the current range of int16_t, so here, IntSet will automatically upgrade the encoding to find the appropriate size.
Therefore, its general process of expansion is as follows:
- First upgrade the encoding to INTSET_ENC_INT32, each integer occupies 4 bytes, and expand the array according to the new encoding method and the number of elements.
- When expanding, use reverse order to copy the elements in the array to the correct position after expansion.
- Then, place the element to be added at the end of the array.
- Finally, change the encoding property of the inset to INTSET_ENC_INT32 and the length property to 4.
Finally, modify the header information of the structure.
Let's take a look at the source code when adding: (I have made notes in the code)
New data flow:
intset *intsetAdd(intset *is, int64_t value, uint8_t *success) {
uint8_t valenc = _intsetValueEncoding(value);//获取当前值的编码
uint32_t pos;//要插入的位置
if (success) *success = 1;
//编码是否超过当前的 intset 的编码
if (valenc > intrev32ifbe(is->encoding)) {
//超出了编码规定，需要升级，调用intsetUpgradeAndAdd具体实现
return intsetUpgradeAndAdd(is,value);
}
// 当时添加的数据还没有超过
else {
// 在当前 intset 中查找值与 value 一样的元素角标
if (intsetSearch(is,value,&pos)) {
if (success) *success = 0;
//若找到了，则无需在执行插入了，直接返回失败
return is;
}
// 数组扩容
is = intsetResize(is,intrev32ifbe(is->length)+1);
// 移动数组中的 pos 之后的所有元素到 pos+1,给新空间腾出空间
if (pos < intrev32ifbe(is->length)) intsetMoveTail(is,pos,pos+1);
}
// 插入新元素
_intsetSet(is,pos,value);
// 重置元素的长度
is->length = intrev32ifbe(intrev32ifbe(is->length)+1);
return is;
}
IntSet upgrade process:
/* Upgrades the intset to a larger encoding and inserts the given integer. */
static intset *intsetUpgradeAndAdd(intset *is, int64_t value) {
//获取当前 intset 的编码
uint8_t curenc = intrev32ifbe(is->encoding);
//获取新编码
uint8_t newenc = _intsetValueEncoding(value);
// 获取元素个数
int length = intrev32ifbe(is->length);
//判断新加入的元素时大于0还是小于0，大于加队尾，小于加队首
int prepend = value < 0 ? 1 : 0;
// 重置编码为新编码
is->encoding = intrev32ifbe(newenc);
// 重置数组大小
is = intsetResize(is,intrev32ifbe(is->length)+1);
// 倒序遍历，逐个搬运旧元素到新的位置，_intsetGetEncoded按照旧编码方式查找旧元素
while(length--)// _intsetSet按照新编码方式插入新元素
_intsetSet(is,length+prepend,_intsetGetEncoded(is,length,curenc));
/* 插入新元素，prepend决定是队首还是队尾*/
if (prepend)
_intsetSet(is,0,value);
else
_intsetSet(is,intrev32ifbe(is->length),value);
// 修改数组长度
is->length = intrev32ifbe(intrev32ifbe(is->length)+1);
return is;
}
In the newly added data, to find out whether the current element already exists, a binary search is used. (The code for binary search is not analyzed in detail, just take a look)
Dict
As we all know, in Redis, it is a database in the form of a key-value pair (key-value pair), so we can quickly add, delete, modify and query according to the key, and the mapping relationship between the key and the value is through Dict. Achieved.
Dict is mainly implemented by three parts:Hash table (DictHashTable), hash node (DictEntry), dictionary (Dict)
Hash table data:
typedef struct dictht {
// entry数组
// 数组中保存的是指向entry的指针
dictEntry **table;
// 哈希表大小
unsigned long size;
// 哈希表大小的掩码，等于的是 size - 1
unsigned long sizemask;
// entry个数
unsigned long used;
} dictht;
Hash node source code:
typedef struct dictEntry {
void *key; // 键
union {
void *val;
uint64_t u64;
int64_t s64;
double d;
} v; // 值
// 下一个Entry的指针
struct dictEntry *next;
} dictEntry;
When we add key-value pairs to Dict, Redis will first calculate the hash value (h) according to the key, and then use h & sizemask AND operation to calculate the index position where the element should be stored in the array.
The source code of the dictionary:
typedef struct dict {
dictType *type; // dict类型，内置不同的hash函数
void *privdata; // 私有数据，在做特殊hash运算时用
dictht ht[2]; // 在一个Dict包含两个哈希表，其中一个是当前数据，另一个一般是空，rehash时使用
long rehashidx; // rehash的进度，-1表示未进行
int16_t pauserehash; // rehash是否暂停，1则暂停，0则继续
} dict;
In Dict, its storage method is illustrated as:
It can be observed here that there are two hash tables in a dictionary, but generally only one hash table is used when it is used, and the other hash table is only used when rehash.
Dict rehash
Expansion in Dict:
The HashTable in Dict is the realization of an array combined with a one-way linked list. When there are many elements in the set, the chance of hash conflict will inevitably increase. When the linked list is too long, its query efficiency will inevitably decrease.
Therefore, the load factor (LoadFactor = used / size) will be checked every time a new key-value pair is added in Dict , and the following two conditions are met. It will trigger the mechanism of hash expansion.
- The LoadFactor of the hash table is >= 1, and the server does not execute background processes such as BGSAVE or BGREWRITEAOF.
- LoadFactor > 5 for hash tables
Analyze the source code:
static int _dictExpandIfNeeded(dict *d){
// 如果正在rehash，则返回ok
if (dictIsRehashing(d)) return DICT_OK; // 如果哈希表为空，则初始化哈希表为默认大小：4
if (d->ht[0].size == 0) return dictExpand(d, DICT_HT_INITIAL_SIZE);
// 当负载因子（used/size）达到1以上，并且当前没有进行bgrewrite等子进程操作
// 或者负载因子超过5，则进行 dictExpand ，也就是扩容
if (d->ht[0].used >= d->ht[0].size &&
(dict_can_resize || d->ht[0].used/d->ht[0].size > dict_force_resize_ratio){
// 扩容大小为used + 1，底层会对扩容大小做判断，实际上找的是第一个大于等于 used+1 的 2^n
return dictExpand(d, d->ht[0].used + 1);
}
return DICT_OK;
}
Of course, in addition to over-expansion, a judgment on the responsible factor will also be made during shrinkage. When LoadFactor < 0.1, hash shrinkage will occur.
Analyze the source code:
// t_hash.c # hashTypeDeleted()
...
if (dictDelete((dict*)o->ptr, field) == C_OK) {
deleted = 1;
// 删除成功后，检查是否需要重置Dict大小，如果需要则调用dictResize重置 /* Always check if the dictionary needs a resize after a delete. */
if (htNeedsResize(o->ptr)) dictResize(o->ptr);
}
...
// server.c 文件
int htNeedsResize(dict *dict) {
long long size, used;
// 哈希表大小
size = dictSlots(dict);
// entry数量
used = dictSize(dict);
// size > 4（哈希表初识大小）并且 负载因子低于0.1
return (size > DICT_HT_INITIAL_SIZE && (used*100/size < HASHTABLE_MIN_FILL));
}
int dictResize(dict *d){
unsigned long minimal;
// 如果正在做bgsave或bgrewriteof或rehash，则返回错误
if (!dict_can_resize || dictIsRehashing(d))
return DICT_ERR;
// 获取used，也就是entry个数
minimal = d->ht[0].used;
// 如果used小于4，则重置为4
if (minimal < DICT_HT_INITIAL_SIZE)
minimal = DICT_HT_INITIAL_SIZE;
// 重置大小为minimal，其实是第一个大于等于minimal的2^n
return dictExpand(d, minimal);
As mentioned above, when BGSAVE or BGREWRITEAOF occurs in Redis, it will temporarily prevent rehash from occurring.
What exactly is rehash?
Regardless of expansion or contraction, a new hash table must be created, resulting in changes in the size and sizemask of the hash table, and the query of the key is related to the sizemask. Therefore, it is necessary to recalculate the index for each key in the hash table and insert a new hash table. This process is called rehash.
The general process of rehash is as follows:
- First calculate the realeSize of the new hash table, the value depends on whether the current expansion or contraction is to be done:
If it is expansion, the new size is the first 2^n greater than or equal to dict.ht[0].used + 1.
If it is contraction, the new size is the first 2^n greater than or equal to dict.ht[0].used (minimum not less than 4)
- Apply for memory space according to the new realeSize, create dicttht, and assign it to dict.ht[1];
- Set dict.rehashidx = 0 to mark the start of rehash;
- Rehash each dictEntry in dict.ht[0] to dict.ht[1];
- at lastAssign dict.ht[1] to dict.ht[0] and initialize dict.ht[1] to an empty hash table, release the memory of the original dict.ht[0].
In this process, the default value (-1) of rehashidx in the dict will change to 0 when rehash starts, and after the final rehash is completed, the value of rehashidx will be changed to -1 again.
But here we also need to consider a problem, Redis is single-threaded, if the amount of data is too large during rehash, it will cause the thread to enter a blocked state, which is extremely bad for our experience.
So the rehash in Dict is never done once. Therefore, up to now, rehash is divided into multiple times and completed gradually. Hence it is also called progressive rehash .
And its specific process is as follows:
- Calculate the size of the new hash table, the value depends on whether the current expansion or contraction is to be done:
If it is expansion, the new size is the first 2^n greater than or equal to dict.ht[0].used + 1.
If it is contraction, the new size is the first 2^n greater than or equal to dict.ht[0].used (minimum not less than 4)
- Apply for memory space according to the new size, create dicttht, and assign it to dict.ht[1]
- Set dict.rehashidx = 0 to mark the start of rehash
- Every time you perform a new, query, modify, or delete operation, check whether dict.rehashidx is greater than -1, and if so, rehash the entry list of dict.ht[0].table[rehashidx] to dict.ht[1 ], and will rehashidx++ . All data up to dict.ht[0] are rehash to dict.ht[1]
- Assign dict.ht[1] to dict.ht[0], initialize dict.ht[1] to an empty hash table, and release the memory of the original dict.ht[0]
- Finally, assign rehashidx to -1, which means the end of rehash
- In the process of rehash, new operations are directly written into ht[1], query, modification and deletion are searched and executed in dict.ht[0] and dict.ht[1] in turn. This can ensure that the data of ht[0] will only decrease but not increase, and will eventually be empty with rehash
This also explains why rehashidx in Dict defaults to -1 instead of 0 as we often think. In the process of rehash, we will use the rehashidx seat array subscript
DictSummary
The structure of Dict:
It is similar to java's HashTable. The bottom layer is an array plus a linked list to resolve hash conflicts.
Dict contains two hash tables. ht[0] is usually used, and ht[1] is used for rehash
Scaling of Dict:
When LoadFactor is greater than 5 or LoadFactor is greater than 1 and there are no child process tasks, Dict expands
The expansion size is the first 2^n greater than or equal to used + 1
Dict shrinks when LoadFactor is less than 0.1
The shrinkage size is the first 2^n greater than or equal to used
Dict adopts progressive rehash, and rehash is executed every time Dict is accessed; it is guaranteed that ht[0] will only decrease but not increase during rehash, new operations are only performed on ht[1], and other operations are performed between the two hashes
ZipList
ZipList is a special kind of "double-ended linked list", which is composed of a series of specially encoded contiguous memory blocks. A push/pop operation can be done on either end, and the time complexity of this operation is O(1).
The basic mechanism of ZipList is:
| Attributes | type | length | use |
|---|---|---|---|
| zlbytes | uint32_t | 4 bytes | Record the number of memory words occupied by the entire compressed list |
| gilded | uint32_t | 4 bytes | It records the number of bytes between the end node of the compressed list and the start address of the compressed list. Through this offset, the address of the end node of the table can be determined. |
| zllen | uint16_t | 2 bytes | Records the number of nodes contained in the compressed list. The maximum value is UINT16_MAX (65534). If it exceeds this value, it will be recorded as 65535 here, but the actual number of nodes needs to traverse the entire compressed list to be calculated. |
| entry | list node | indefinite | Each node contained in the compressed list, the length of the node is determined by the content stored in the node. |
| zlend | uint8_t | 1 byte | Special value 0xFF (255 decimal), used to mark the end of the compressed list. |
The Entry in ZipList does not record the pointers of front and back nodes like ordinary linked lists, because recording two pointers takes up 16 bytes, wasting memory.
So another structure is adopted in ZipList.
- previous_entry_length: The length of the previous node, 1 or 5 bytes.
If the length of the previous node is less than 254 bytes, use 1 byte to save the length value;
if the length of the previous node is greater than 254 bytes, use 5 bytes to save the length value, the first byte It is 0xfe, the last four bytes are the real length data
- encoding: encoding attribute, record the data type of content (string or integer) and length, occupying 1, 2 or 5 bytes
- contents: responsible for saving the data of the node, which can be a string or an integer
Notice:
All values of storage length in ZipList adopt little-endian byte order, that is, the low-order byte comes first, and the high-order byte follows. For example: the value 0x1234, the actual storage value after adopting little-endian byte order: 0x3412
Issues with chained updates in ZipList
Each Entry in ZipList contains previous_entry_length to record the size of the previous node, the length is 1 or 5 bytes:
If the length of the previous node is less than 254 bytes, use 1 byte to save the length value;
if the length of the previous node is greater than or equal to 254 bytes, use 5 bytes to save the length value, the first character The section is 0xfe, and the last four bytes are the real length data
Now, suppose we have N consecutive entries with a length between 250 and 253 bytes, so the previous_entry_length attribute of the entry can be represented by 1 byte.
Now we add a new entry data size of 254bytes in the header position. Then according to the judgment of pre_entry_len in the following entry, the size is changed to 5 bytes, then the sum of the following entries is greater than or equal to 254, and the size of pre_entry_len also needs to be changed for the subsequent entries, and so on.
The continuous multiple space expansion operations generated in this special case of ZipList are calledCascade Update. Addition and deletion may lead to chain updates.
ZipList features:
- The compressed list can be regarded as a "doubly linked list" of continuous memory space
- The nodes in the list are not connected by pointers, but are addressed by recording the length of the previous node and this node, and the memory usage is low
- If there is too much data in the list, the linked list will be too long, which may affect the query performance.
- Continuous update problems may occur when adding or deleting large data.
QuickList
QuickList is used to store large amounts of data.
There are still some problems in the ZipList above:
Question 1: Although ZipList saves memory, the application memory must be a continuous space. If the memory occupies a lot, the application memory efficiency is very low. what to do?
To alleviate this problem, we must limit the length and entry size of the ZipList.
Question 2: But what should we do if we want to store a large amount of data and exceed the optimal upper limit of ZipList?
We can create multiple ZipLists to store data in pieces.
Question 3: After the data is split, it is scattered and inconvenient to manage and search. How do these multiple ZipLists establish a relationship?
Redis introduced a new data structure QuickList in version 3.2, which is a double-ended linked list, except that each node in the linked list is a ZipList.
In order to avoid too many entries in each ZipList in QuickList, Redis provides a configuration item: list-max-ziplist-size to limit.
- If the value is positive, it represents the maximum number of entries allowed in ZipList
- If the value is negative, it represents the maximum memory size of ZipList, divided into 5 cases:
-1: The memory usage of each ZipList cannot exceed 4kb
-2: The memory usage of each ZipList cannot exceed 8kb
-3: The memory usage of each ZipList Occupancy cannot exceed 16kb
-4: The memory occupation of each ZipList cannot exceed 32kb
-5: The memory occupation of each ZipList cannot exceed 64kb
, the default value is -2;
In addition to controlling the size of ZipList in Redis, QuickList can alsoZipList does compression. It is controlled by the configuration item list-compress-depth. Because linked lists are generally accessed more from the beginning to the end, the beginning and the end are not compressed. This parameter is to control the number of nodes that are not compressed at the beginning and end:
- 0: special value, representing no compression
- 1: Indicates that the first and last nodes of the QuickList are not compressed, and the middle nodes are compressed
- 2: Indicating that the first and last two nodes of the QuickList are not compressed, the middle nodes are compressed,
and so on.
The default value is 0;
source code:
typedef struct quicklist {
// 头节点指针
quicklistNode *head;
// 尾节点指针
quicklistNode *tail;
// 所有ziplist的entry的数量
unsigned long count;
// ziplists总数量
unsigned long len;
// ziplist的entry上限，默认值 -2
int fill : QL_FILL_BITS; // 首尾不压缩的节点数量
unsigned int compress : QL_COMP_BITS;
// 内存重分配时的书签数量及数组，一般用不到
unsigned int bookmark_count: QL_BM_BITS;
quicklistBookmark bookmarks[];
} quicklist;
typedef struct quicklistNode {
// 前一个节点指针
struct quicklistNode *prev;
// 下一个节点指针
struct quicklistNode *next;
// 当前节点的ZipList指针
unsigned char *zl;
// 当前节点的ZipList的字节大小
unsigned int sz;
// 当前节点的ZipList的entry个数
unsigned int count : 16;
// 编码方式：1，ZipList; 2，lzf压缩模式
unsigned int encoding : 2;
// 数据容器类型（预留）：1，其它；2，ZipList
unsigned int container : 2;
// 是否被解压缩。1：则说明被解压了，将来要重新压缩
unsigned int recompress : 1;
unsigned int attempted_compress : 1; //测试用
unsigned int extra : 10; /*预留字段*/
} quicklistNode;
Illustration:
Features of QuickList:
- Is a double-ended linked list whose node is ZipList
- The node adopts ZipList, which solves the memory occupation problem of the traditional linked list
- Control the size of ZipList to solve the problem of continuous memory space application efficiency
- Intermediate nodes can be compressed, further saving memory
ShipList
SkipList (jump list) is essentially a linked list. And it is a doubly linked list,
it also has some of its own characteristics:
- elements are stored in ascending order
- Nodes may contain multiple pointers with different pointer spans.
Simple concept map:
Take a look at its source code:
// t_zset.c
typedef struct zskiplist {
// 头尾节点指针
struct zskiplistNode *header, *tail;
// 节点数量
unsigned long length;
// 最大的索引层级，默认是1
int level;
} zskiplist;
// t_zset.c
typedef struct zskiplistNode {
sds ele; // 节点存储的值
double score;// 节点分数，排序、查找用
struct zskiplistNode *backward; // 前一个节点指针
struct zskiplistLevel {
struct zskiplistNode *forward; // 下一个节点指针
unsigned long span; // 索引跨度
} level[]; // 多级索引数组
} zskiplistNode;
Features of SkipList:
- The jump table is a doubly linked list, each node contains score and ele values
- Nodes are sorted according to the score value, and if the score value is the same, they are sorted according to the ele dictionary
- Each node can contain multiple layers of pointers, and the number of layers is a random number between 1 and 32
- Different layer pointers have different spans to the next node, the higher the layer, the larger the span
- The efficiency of adding, deleting, modifying, and checking is basically the same as that of the red-black tree, but the implementation is simpler
RedisObject
RedisObject is the key and value of any data type in Redis will be encapsulated into a RedisObject, also called Redis object,
The source code is as follows:
here is just the header information of RedisObject.
typedef struct redisObject {
unsigned type:4;
unsigned encoding:4;
unsigned lru:LRU_BITS;
int refcount;
void *ptr;
} robj;
unsigned type: 4; Indicates the object type, which are String, hash, list, set and zset, occupying 4 bits.
#define OBJ_STRING 0
#define OBJ_LIST 1
#define OBJ_SET 2
#define OBJ_ZSET 3
#define OBJ_HASH 4
unsigned encoding:4; Indicates the underlying encoding method, there are 11 types in total, occupying 4 bits
unsigned lru:LRU_BITS; Indicates the last time the object was accessed, which occupies 24 bits, which is convenient for judging the key that has been idle for too long.
int refcount; Indicates the object reference counter, if the counter is 0, it means that the object is unreferenced and can be recycled.
void *ptr;: Indicates a pointer, pointing to the space where the actual data is stored.
In Redis, different encoding methods will be selected according to the different types of stored data, including 11 different types in total:
| serial number | Encoding | illustrate |
|---|---|---|
| 0 | OBJ_ENCODING_RAW | raw encoding dynamic string |
| 1 | OBJ_ENCODING_INT | a string of integers of type long |
| 2 | OBJ_ENCODING_HT | hash table (dictionary dict) |
| 3 | OBJ_ENCODING_ZIPMAP | obsolete |
| 4 | OBJ_ENCODING_LINKEDLIST | double-ended linked list |
| 5 | OBJ_ENCODING_ZIPLIST | compressed list |
| 6 | OBJ_ENCODING_INTSET | set of integers |
| 7 | OBJ_ENCODING_SKIPLIST | jump table |
| 8 | OBJ_ENCODING_EMBSTR | dynamic string for embstr |
| 9 | OBJ_ENCODING_QUICKLIST | quick list |
| 10 | OBJ_ENCODING_STREAM | flow |
In Redis, different encoding methods will be selected according to the type of data stored. The encoding used for each data type is as follows:
| type of data | Encoding |
|---|---|
| OBJ_STRING | int、embstr、raw |
| OBJ_LIST | LinkedList and ZipList (before 3.2), QuickList (after 3.2) |
| OBJ_SET | intset、HT |
| OBJ_ZSET | ZipList、HT、SkipList |
| OBJ_HASH | ZipList、HT |
Next article: ===》 A detailed look at the five data types in Redis from the bottom layer