Redis 6.0 source code reading notes (3)-Overview of the important data structures of Redis and its 6 data types

1. Important data structure

There are many important Redis data structures, which storage structure from the outer to the inner layer followed by redisDb, dict, dictht,dictEntry

There are 16 redisDbs by default. Each redisDb contains a dict data structure. The dict contains a dictht array. The number of arrays is 2, which is mainly used for hash expansion. dictht contains an array of internal dictEntry, dictEntry is actually a key-value node hash table, if the conflict chain address law to solve

1.1 redisServer

Data structure redisServeris a redis server abstraction, as defined in server.hthe. redisServerThere are a lot of attributes in, the following is part of an excerpt, a brief introduction

1. hz: Redis timing task trigger frequency
2. *db: redisDb array, default 16 redisDb
3. *commands: A dictionary of commands supported by redis
4. *el: Redis event loop example
5. runid[CONFIG_RUN_ID_SIZE+1]: Runid of the current redis instance

struct redisServer {
    
    
    /* General */
    pid_t pid;                  /* Main process pid. */
    
    ......
    
    int hz;                     /* serverCron() calls frequency in hertz */
    redisDb *db;
    dict *commands;             /* Command table */
    dict *orig_commands;        /* Command table before command renaming. */
    aeEventLoop *el;
    
    ......
    
    char runid[CONFIG_RUN_ID_SIZE+1];  /* ID always different at every exec. */
    
    ......
    
    list *clients;              /* List of active clients */
    list *clients_to_close;     /* Clients to close asynchronously */
    list *clients_pending_write; /* There is to write or install handler. */
    list *clients_pending_read;  /* Client has pending read socket buffers. */
    list *slaves, *monitors;    /* List of slaves and MONITORs */
    client *current_client;     /* Current client executing the command. */
  
    ......
};

1.2 redisDb

redisDbAbstract redis database definition server.h, the more critical properties are as follows

1. *dict: Dictionary to store data
2. *expires: Store the key with expiration time

typedef struct redisDb {
    
    
    dict *dict;                 /* The keyspace for this DB */
    dict *expires;              /* Timeout of keys with a timeout set */
    dict *blocking_keys;        /* Keys with clients waiting for data (BLPOP)*/
    dict *ready_keys;           /* Blocked keys that received a PUSH */
    dict *watched_keys;         /* WATCHED keys for MULTI/EXEC CAS */
    int id;                     /* Database ID */
    long long avg_ttl;          /* Average TTL, just for stats */
    unsigned long expires_cursor; /* Cursor of the active expire cycle. */
    list *defrag_later;         /* List of key names to attempt to defrag one by one, gradually. */
} redisDb;

1.3 dict

dictRedis is in the dictionary, the definition in the dict.hfile, its main attributes are as follows

1. ht[2]: Hash table array, for expansion convenience, there are 2 elements, one of the hash tables normally stores data, the other hash table is empty, the empty hash table is used when rehashing
2. rehashidx:rehash index, when rehash is not in progress, the value is -1

typedef struct dict {
    
    
    dictType *type;
    void *privdata;
    dictht ht[2];
    long rehashidx; /* rehashing not in progress if rehashidx == -1 */
    unsigned long iterators; /* number of iterators currently running */
} dict;

1.4 dictht

dicthtA hash table structure is defined in the dict.hfile, the following important properties

1. **table: key-value key-value pair node array, similar to the underlying array of HashMap in Java
2. size: Hash table capacity size
3. sizemask: Always equal to size-1, used to calculate the index value
4. used: The number of dictEntry actually stored in the hash table

typedef struct dictht {
    
    
    dictEntry **table;
    unsigned long size;
    unsigned long sizemask;
    unsigned long used;
} dictht;

1.5 dictEntry

dictEntryIs a key-value pair in the key redis node, the node data is actually stored is defined in the dict.hfile, the following important properties

1. *key: Key object, always a string type object
2. *val: Value object, which may be any type of object
3. *next: Tail pointer, point to the next node

typedef struct dictEntry {
    
    
    void *key;
    union {
    
    
        void *val;
        uint64_t u64;
        int64_t s64;
        double d;
    } v;
    struct dictEntry *next;
} dictEntry;

2. Data Type

2.1 Redis data object structure

All data in the Redis database isStore as key-value node dictEntryWherein a key and value are redisObjectstructure objects, but always a key string type objects, the object value may be any type of data. redisObjectIt is defined in the structure server.hshown below

You can see that the important attributes in the structure are as follows,Different objects have different types, and the same type of type will have different storage forms encoding

1. type: This attribute indicates the type of data object, such as String, List, etc.
2. encoding: This attribute indicates the underlying storage structure of the object. For example, the possible storage structures of ZSet type objects are ZIPLIST and SKIPLIST
3. *ptr: Pointer to the underlying storage structure

typedef struct redisObject {
    
    
    unsigned type:4;
    unsigned encoding:4;
    unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or
                            * LFU data (least significant 8 bits frequency
                            * and most significant 16 bits access time). */
    int refcount;
    void *ptr;
} robj;

2.2 Redis data types and storage structure

Redis illustrated above in relation to the data type and its storage configuration, FIG macros in the server.hfile

/* The actual Redis Object */
#define OBJ_STRING 0    /* String object. */
#define OBJ_LIST 1      /* List object. */
#define OBJ_SET 2       /* Set object. */
#define OBJ_ZSET 3      /* Sorted set object. */
#define OBJ_HASH 4      /* Hash object. */

/* The "module" object type is a special one that signals that the object
 * is one directly managed by a Redis module. In this case the value points
 * to a moduleValue struct, which contains the object value (which is only
 * handled by the module itself) and the RedisModuleType struct which lists
 * function pointers in order to serialize, deserialize, AOF-rewrite and
 * free the object.
 *
 * Inside the RDB file, module types are encoded as OBJ_MODULE followed
 * by a 64 bit module type ID, which has a 54 bits module-specific signature
 * in order to dispatch the loading to the right module, plus a 10 bits
 * encoding version. */
#define OBJ_MODULE 5    /* Module object. */
#define OBJ_STREAM 6    /* Stream object. */



/* Objects encoding. Some kind of objects like Strings and Hashes can be
 * internally represented in multiple ways. The 'encoding' field of the object
 * is set to one of this fields for this object. */
#define OBJ_ENCODING_RAW 0     /* Raw representation */
#define OBJ_ENCODING_INT 1     /* Encoded as integer */
#define OBJ_ENCODING_HT 2      /* Encoded as hash table */
#define OBJ_ENCODING_ZIPMAP 3  /* Encoded as zipmap */
#define OBJ_ENCODING_LINKEDLIST 4 /* No longer used: old list encoding. */
#define OBJ_ENCODING_ZIPLIST 5 /* Encoded as ziplist */
#define OBJ_ENCODING_INTSET 6  /* Encoded as intset */
#define OBJ_ENCODING_SKIPLIST 7  /* Encoded as skiplist */
#define OBJ_ENCODING_EMBSTR 8  /* Embedded sds string encoding */
#define OBJ_ENCODING_QUICKLIST 9 /* Encoded as linked list of ziplists */
#define OBJ_ENCODING_STREAM 10 /* Encoded as a radix tree of listpacks */

2.2.1 String object String

OBJ_STRING The underlying data structure of string objects is generallySimple Dynamic String (SDS), But its storage method can be OBJ_ENCODING_INT, OBJ_ENCODING_EMBSTRand OBJ_ENCODING_RAW, different storage methods represent different object memory structures

1. OBJ_ENCODING_INT
   If the saved stringThe length is less than 20 and can be parsed as an integer, Then this integer will be saved directly in redisObjectthe ptrproperty in
2. OBJ_ENCODING_EMBSTR
   Strings less than 44 (OBJ_ENCODING_EMBSTR_SIZE_LIMIT) will beSimple Dynamic String (SDS) The form of storage, but the malloc method will be used to allocate memory once, and the redisObject object header and SDS object will continue to exist together
3. OBJ_ENCODING_RAW
   The string will start withSimple Dynamic String (SDS) The form of storage requires two malloc to allocate memory. The redisObject object header and the SDS object are generally discontinuous in the memory address

• In Redis, long and double floating-point numbers are encapsulated in string objects and then stored
• OBJ_ENCODING_RAWThe OBJ_ENCODING_EMBSTRdifference is that the memory allocation and release, raw requires two operations, embstr only once, since addition embstr continuous memory block, to better utilize the cache benefits of
• OBJ_ENCODING_EMBSTR The encoded object is read-only, once modified, it will be transcoded intoOBJ_ENCODING_RAW

2.2.2 List Object List

OBJ_LISTThere have been three implementations of the underlying storage structure of the list object, namely OBJ_ENCODING_LINKEDLIST, OBJ_ENCODING_ZIPLISTand OBJ_ENCODING_QUICKLIST,Among them, OBJ_ENCODING_LINKEDLIST is obsolete after version 3.2

1. OBJ_ENCODING_LINKEDLIST
   The bottom layer is implemented by a double-ended linked list, each linked list node stores a string object, and an element is stored in each string object
2. OBJ_ENCODING_ZIPLIST
   The underlying implementation is similar to an array, using characteristic attributes to store the meta information of the entire list, such as the memory size occupied by the entire list, the start position of the data saved in the list, the number of data saved in the list, etc. The saved data is encapsulated in zlentry
   Two conditions need to be met to use ziplist
   • The length of all string elements saved by the list object is less than 64 bytes
   • The number of elements stored in the list object is less than 512
3. OBJ_ENCODING_QUICKLIST
   The bottom layer uses a double-ended linked list structure, but each linked list node saves a ziplist, and the data is stored in the ziplist

2.2.3 Set

OBJ_SETThere are two underlying storage structures for collection objects, OBJ_ENCODING_HTandOBJ_ENCODING_INTSET

1. OBJ_ENCODING_INTSET
   All elements stored in the collection are integer values. This storage structure will be used, but when the number of elements stored in the collection object exceeds512 (configured by server.set_max_intset_entries)Will be converted to OBJ_ENCODING_HT
2. OBJ_ENCODING_HT
   The bottom of dictthe dictionary, the dictionary data as a save key, corresponding to the key value is NULL, and the like in Java HashSet

2.2.4 ZSet

OBJ_ZSETStorage structure into an ordered set of objects OBJ_ENCODING_SKIPLISTandOBJ_ENCODING_ZIPLIST

1. OBJ_ENCODING_ZIPLIST
   When ziplist is used as the underlying storage structure of zset, each set element is stored using two compressed list nodes next to each other. The first node stores the element value, and the second element stores the element’s score, and the score is small Close to the head of the table, large close to the end of the table

• Ordered collection objects need to be stored using ziplistMeet the following two conditions at the same time, do not meet any one of the conditions will use skiplist
  • All elements are less than 64 bytes in length (server.zset_max_ziplist_value configuration)
  • The number of elements is less than 128 (server.zset-max-ziplist-entries configuration)

2. OBJ_ENCODING_SKIPLIST
   The underlying implementation is a jump table combined with a dictionary. Each jump table node saves a collection element, which is arranged in descending order of score. The object attribute of the node saves the value of the element, and the score attribute saves the score; each key-value pair of the dictionary saves a collection element, the element value Packed as dictionary keys, element scores are saved as dictionary values

• skiplist 同时使用跳跃表和字典实现的原因
  • The advantage of skip table is order, but the complexity of query score is O(logn); the complexity of dictionary query score is O(1), but disorder, combining the two can achieve complementary advantages
  • The element members and scores of the set are shared, and the jump table and dictionary point to the same address through pointers, so memory is not wasted

2.2.5 Hash object

OBJ_HASHThe structure is divided into storage OBJ_ENCODING_ZIPLISTand OBJ_ENCODING_HTwhich achieve the following:

1. OBJ_ENCODING_ZIPLIST
   In a hash object that stores data in a ziplist structure, the key-value key-value pairs are stored in the compressed linked list in a closely connected manner. The key is first placed at the end of the table, and then the value; the key-value pairs are always added to the end of the table

• Hash objects need to use ziplist to store dataMeet the following two conditions at the same time, do not meet any of them, use the dict structure
  • The length of the key and value strings of all key-value pairs is less than 64 (server.hash_max_ziplist_value configuration) bytes
  • The number of key-value pairs is less than 512 (server.hash-max-ziplist-entries)

2. OBJ_ENCODING_HT
   Bottom for dictthe dictionary, each of the objects hash key-value pairs of key uses a dictionary dictEntryto store, dictionary keys and values are String object

2.2.6 Stream Object Stream

OBJ_ENCODING_STREAMThe stream object is a new data object introduced in version 5.0. It is very similar to the list object List, but with more powerful functions, with the ability to subscribe and publish. Storage structure its use OBJ_ENCODING_STREAMwith other storage structures different

OBJ_ENCODING_STREAM
Underlying the use of compression prefix tree (radix tree) to store the string elements, from the source file rax.hcomments can know, radix tree is actually a dictionary tree (Trie tree) compression optimized version, it willCompress the characters saved by multiple consecutive nodes with only one child node into one node

• The principle of Trie Tree
  splits each string element key according to a single character, and then corresponds to each branch. In this way, it traverses from the root node to the leaf node of a branch, and the string of characters saved by all the passed nodes is the element key corresponding to this branch.

Trie tree ：
 *              (f) ""
 *                \
 *                (o) "f"
 *                  \
 *                  (o) "fo"
 *                    \
 *                  [t   b] "foo"
 *                  /     \
 *         "foot" (e)     (a) "foob"
 *                /         \
 *      "foote" (r)         (r) "fooba"
 *              /             \
 *    "footer" []             [] "foobar"


Radix tree:
                   ["foo"] ""
 *                     |
 *                  [t   b] "foo"
 *                  /     \
 *        "foot" ("er")    ("ar") "foob"
 *                 /          \
 *       "footer" []          [] "foobar"