One: Objects in Redis

Redis uses objects to represent the keys and values in the database. Whenever we create a new key-value pair in the Redis database, we will create at least two objects. One object is used as the key of the key-value pair (key object). Another object is used as the value of the key-value pair (value object).

Each object in Redis is represented by a redisObject structure. The three attributes related to the saved data in this structure are the type attribute, the encoding attribute, and the ptr attribute.

typedef struct redisObject {
    //类型
    unsigned type:4;
    //编码
    unsigned encoding:4;
    //最近访问的时间
    unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or
                            * LFU data (least significant 8 bits frequency
                            * and most significant 16 bits access time). */
    //引用计数
    int refcount;
    //指向底层实现数据结构的指针
    void *ptr;
} robj;

1: type

There are five types of objects:

The key-value pair saved by redis, the key is always a string object, and the value can be one of string objects, list objects, hash objects, collection objects, and ordered collection objects, so:

When we call a database key a string object, the value corresponding to the database key we represent is a string object.
When we call a database key a list object, the value corresponding to the database key we represent is a list object.

2: encoding and underlying implementation

The object's ptr pointer points to the underlying implementation data structure of the object, and these data structures are determined by the encoding attribute of the object.

The encoding attribute records the encoding used by the object, which means that according to this encoding, you can know what data structure the object uses as the underlying data structure

At least two different encodings are used for each type of object

Set the encoding used by the object through the encoding attribute, instead of a fixed encoding associated with a specific type of object, which greatly improves the flexibility and efficiency of Redis, because Redis can set different settings for an object according to different scenarios Encoding to optimize the efficiency of the object in a certain scene.

3: String object

The encoding of the string object can be int, raw or embstr

If a string object saves an integer value, and the integer value can be represented by the long type, then the string object will save the integer value in the ptr attribute of the string object structure (convert void* to long) , And set the encoding of the string object to int.

If a string object stores a string, and the length of the string is greater than 32 bytes, then the string object will use a simple dynamic string (SDS) to save the value of the string, and the character The encoding of the string object is set to raw.

If a string object stores a string, and the length of the string is less than or equal to 32 bytes, then the string object will use embstr encoding to save the string value.

Embstr encoding is an optimized encoding method specially used to save short strings. This encoding, like raw encoding, uses the redisObject structure and sdshdr structure, while embstr encoding allocates a continuous space by calling a memory allocation function , Space species includes redisObject structure and sdshdr structure.

The advantages of embstr over raw are:

The number of memory allocations required for embstr encoding to create a string object has changed from two to one in raw.
To release an embstr-encoded string object only needs to call the memory release function once, while releasing a raw-encoded string object needs to call the memory release function twice.
All the data of the embstr encoded string is in a contiguous memory, so this encoded string object can better utilize the advantages of the cache than the raw encoded string object.

The floating-point number represented by the long double type is also stored as a string value in redis.

When saving, we will convert the floating point value to a string value to save
When performing operations, we will convert the string value into a floating point value, and then perform the operation, the result obtained is converted into a string value, and then saved.

3.1: Conversion of string object encoding

Int-encoded string objects and embstr-encoded string objects will be converted to raw-encoded strings if the conditions are met.

For int-encoded objects, if we perform a series of operations so that the value saved by the object is no longer an integer value, but a string, then the encoding of the string object will change from int to raw.
The embstr-encoded string is actually read-only. When we execute the modification command on the embstr-encoded string object, the program will first change the encoding of the string object to raw, and then perform the modification operation.

3.2: Implementation of string commands

4: List object

The encoding of the list object can be ziplist or linkedlist

The ziplist-encoded list object uses a compressed list as the underlying implementation, and each compressed list node (entry) stores a list element.

The ziplist-encoded list object uses a double-ended linked list as the bottom layer. Each double-ended list node stores a string object, and each string object stores a list element

4.1: Encoding conversion

The length of all string elements stored in the list object is less than 64 bytes
The number of elements stored in the list object is less than 512, and the list object that cannot meet these two conditions needs to use linkedlist encoding

list-max-ziplist-value 64
list-max-ziplist-entries 512

4.2: Implementation of List Command

5: Hash object

The encoding of the hash object can be ziplist or hashtable.

The hash object encoded by ziplist is implemented with the compressed list as the bottom layer. Whenever there is a new key-value pair to be added to the hash object, the program will first push the compressed list node of the saved key to the end of the compressed list, and then Then push the compressed list node with the saved value to the end of the compressed list table.

Therefore, it has the following characteristics:

1: Two nodes that save the same key-value pair are always close to each other, the node that saves the key is first, and the node that saves the value is later.

2: The key-value pairs first added to the hash object will be placed at the head of the compressed list, and the key-value pairs added later to the hash object will be placed at the end of the compressed list.

The hash object encoded by hashtable uses a dictionary as the underlying implementation, and each key-value pair in the hash object is stored using the key-value pair of the dictionary.

Each key in the dictionary is a string object, and the value of the key-value pair is stored in the object
Each value in the dictionary is a string object, and the value of the key-value pair is stored in the object

5.1: Character encoding conversion

The length of the key and value strings of all key-value pairs saved by the hash object is less than 64 bytes
The number of key-value pairs saved by the hash object is less than 512, and the hash object that cannot meet these two conditions needs to use hashtable encoding

# Hashes are encoded using a memory efficient data structure when they have a
# small number of entries, and the biggest entry does not exceed a given
# threshold. These thresholds can be configured using the following directives.
hash-max-ziplist-entries 512
hash-max-ziplist-value 64

5.2: Implementation of Hash Command

6: Collection objects

The encoding of the collection object can be intset or hashtable.

The intset-encoded collection object uses the integer collection as the underlying implementation, and all the elements contained in the collection object are stored in the integer collection.
The hashtable-encoded collection object uses a dictionary as the underlying implementation. Each key of the dictionary is a string object, each string object contains a collection element, and the values of the dictionary are all set to NULL.

6.1: Encoding conversion

All elements saved by the collection object are integer values
The number of elements saved by the collection object does not exceed 512. Collection objects that cannot meet these two conditions need to use hashtable encoding

# Sets have a special encoding in just one case: when a set is composed
# of just strings that happen to be integers in radix 10 in the range
# of 64 bit signed integers.
# The following configuration setting sets the limit in the size of the
# set in order to use this special memory saving encoding.
set-max-intset-entries 512

6.2: Implementation of collective commands

7: Ordered collection of objects

The encoding of the collection object can be intset or skiplist.

The hash object encoded by ziplist is implemented with the compressed list as the bottom layer. Each set element is saved by two compressed list nodes next to each other. The first node saves the members of the element, and the second node saves the element’s score. value.

The scores of the set elements of the compressed list are sorted from small to large. Elements with smaller scores are closer to the head of the table, and elements with larger scores are closer to the end of the table.

The ordered set object encoded by skiplist uses the zset structure as the underlying implementation. A zset structure contains a dictionary and a skip list.

typedef struct zset {
    dict *dict;
    zskiplist *zsl;
} zset;

The dictionary in zset creates a mapping from members to scores for an ordered set. Each key-value pair in the dictionary stores a set element. The key of the dictionary stores the members of the element, and the value of the dictionary stores The score of the element, through the dictionary, you can find the score of a given member in o(1).

The jump table also saves the members and scores of the elements, and they all share the same element members and scores through pointers. No duplicate members and points will be generated, so no extra memory will be wasted.

Note: Elements and points are actually shared.

7.1: Encoding conversion

The length of all element members saved by the ordered collection object is less than 64 bytes
The number of elements saved by the ordered collection object is less than 128, and the ordered collection object that cannot meet these two conditions needs to use skiplist encoding

# Similarly to hashes and lists, sorted sets are also specially encoded in
# order to save a lot of space. This encoding is only used when the length and
# elements of a sorted set are below the following limits:
zset-max-ziplist-entries 128
zset-max-ziplist-value 64

7.2: Implementation of ordered set commands

8: Type checking and command polymorphism

There are two types of commands used by Redis to manipulate keys. One type can be executed for any type of key. One is to execute only on the keys of the characteristic type.

DEL, EXPIRE, RENAME, TYEP, OBJECT and other commands are useful for any key

8.1: Type check

8.2: The realization of polymorphic commands

DEL, EXPIRE, RENAME, TYEP, and OBJECT are for the polymorphism of type, while set is for the polymorphism of encoding.

9: Memory reclamation

When the value of refcount under the redisobject structure becomes 0, the memory occupied by the object will be released.

9.1: API to modify reference counting

incrRefCount function

void incrRefCount(robj *o) {
    if (o->refcount < OBJ_FIRST_SPECIAL_REFCOUNT) {
        o->refcount++;
    } else {
        if (o->refcount == OBJ_SHARED_REFCOUNT) {
            /* Nothing to do: this refcount is immutable. */
        } else if (o->refcount == OBJ_STATIC_REFCOUNT) {
            serverPanic("You tried to retain an object allocated in the stack");
        }
    }
}

decrRefCount function

void decrRefCount(robj *o) {
    if (o->refcount == 1) {
        switch(o->type) {
        case OBJ_STRING: freeStringObject(o); break;
        case OBJ_LIST: freeListObject(o); break;
        case OBJ_SET: freeSetObject(o); break;
        case OBJ_ZSET: freeZsetObject(o); break;
        case OBJ_HASH: freeHashObject(o); break;
        case OBJ_MODULE: freeModuleObject(o); break;
        case OBJ_STREAM: freeStreamObject(o); break;
        default: serverPanic("Unknown object type"); break;
        }
        zfree(o);
    } else {
        if (o->refcount <= 0) serverPanic("decrRefCount against refcount <= 0");
        if (o->refcount != OBJ_SHARED_REFCOUNT) o->refcount--;
    }
}

10: Object sharing

When redis is initialized, a string object from 0 to 9999 is created, so that the object will be shared. When different objects are shared, the refcount will increase.

11: The idling time of the object

The lru attribute of the redisObject object records the time of the last visit, and the time of the last visit can be calculated by subtracting the time of the last visit from the current time.

Objects of Redis Design and Implementation