Redis underlying implementation of five data types

Brief introduction

Redis five five data types called data object; 6 described earlier large data structures, Redis not use these structures to implement the key to the database, but these structures are used to build an object system redisObject; The target system contains five data, strings (string), a list of objects (list), the hash object (the hash), a collection (set) an ordered set of objects and objects (zset); and the bottom layer data encoding the five objects can be ordered oBJECT ENCODING to view it.

redisObject structure

typedef struct redisObject {
    // 类型
    unsigned type:4;
    // 编码
    unsigned encoding:4;
    // 指向底层实现数据结构的指针
    void *ptr;
    // ...
} robj;

redis are key-value pairs stored data, so the object is divided into key objects and value objects, that is a key-value store key-value pairs will create two objects, key objects and value objects.
Key object is always a string object, and the value of the object may be any of the five objects.

• type attribute is the type of object storage, that is, we say the string, list, hash, set, zset of one, you can use the command TYPE key to view.
• encoding attribute record used by the encoding formation, i.e., which data structures used to achieve the underlying object.

The table lists the output corresponding to the underlying code and constants OBJECT ENCODING command, the first three are strings structure

When we deposit key-value and key-value pair is not specified object encoding, but Redis will be set according to the system without the use of an object scene is different encoding, up to save memory, speed up the purpose of the visit speed.

String object (string)

String object underlying data structure implemented as a simple dynamic strings (SDS) and directly stored, but coding mode may be int, raw or embstr, except that different memory structures.

(1) int coding

Is an integer value stored in a string, and this can be formally represented as a long type, then it will be directly stored in ptr attribute redisObject's, the encoded set int, as shown:

(2) raw encoded

Value greater than 32 bytes string stored string, a simple dynamic strings (SDS) structure, and encoding to raw, this time is consistent with the SDS structure memory structure, is twice the number of memory allocation, creating objects redisObject and sdshdr structure, as shown:

(3) embstr encoding

String String stored value less than or equal to 32 bytes, it is also simple to use dynamic strings (SDS structure), but the memory architecture is optimized for a string stored Dunxiao; memory allocation need only once to be completed, a continuous space can be allocated as:

string object summary:

• In Redis, the stored long, double floating point type is then converted to a string stored.
• and encoding raw embstr effect is the same, except that the memory allocation and release, raw twice, once embstr.
• embstr continuous block of memory, the cache can make better use of the advantages to
• int code and appending the string embstr coding and other operations, if done, will be converted to the raw coding condition is satisfied; embstr coding object is read, once the first modification will be transcoded to raw.

List of objects (list)

Before version 3.2: coded list of objects that can be ziplist and one linkedlist.

(1) ziplist encoding

ziplist hash coding Caprice underlying implementation is a list of compression, each compressed node in the list holds a list element.

(2) linkedlist encoding

linkedlist encoding double bottom end of linked list implementation, each of the double ended linked list node keeps a String object, to save a list elements within each string object.

List of objects transcoding:

• A list of objects using ziplist coding need to meet two conditions: First, all the strings are smaller than 64 bytes, the second is less than the number of elements 512, will be used does not satisfy any one of linkedlist coding.
• Digital two conditions may be modified in Redis configuration file, list-max-ziplist-value option and a list-max-ziplist-entries option.
• FIG StringObject is on a string objects mentioned, the only string object is used as a nested object in object five.

After version 3.2: the encoding quicklist list of objects.

quicklist coding

quickList zipList and is a mixture of linkedList, it linkedList press section segmented, each segment zipList used to compact storage, connected in series between a bidirectional pointer plurality zipList.


Redis will be on ziplist compressed storage, use LZF compression algorithm, you can select a compression depth.
quicklist default compression depth is 0, that is not compressed. The actual depth of the compression is determined by the configuration parameter list-compress-depth. To support the rapid push / pop operations, inclusive of two ziplist quicklist not compressed, then the depth is 1. If the depth is 2, it means a first ziplist quicklist end to end and a second end to end ziplist not compressed.
ziplist length:
internal quicklist ziplist single default length is 8k bytes, beyond this number of bytes will be a new starting ziplist.
ziplist length of list-max-ziplist-size determined by the configuration parameters.

Hash object (hash)

Encoded hash object can be ziplist and one hashtable.

(1) ziplist encoding

ziplist encoded hash object underlying implementation is compressed list, the hash coding ziplist subject, key-value by way of the key into the compression closely linked list, the first key into the end of the table, and then into value; key-value is always added to the end of the table.

(2) hashtable encoding

hashtable underlying hash object is achieved dictionary coding, each of the objects hash key-value pairs use a dictionary to store the key-value pair.
Key-value pair that is the dictionary, the dictionary keys and values are string object dictionary to save the key-value key, the value of the stored key-value dictionary of value.

Hash object code conversion:

• Ziplist hash coding object uses need to meet two conditions: key values ​​and string length of the first, all keys are less than 64 bytes; the second is less than the number of key-value pairs 512; does not satisfy a use any encoding hashtable .
• The above two conditions can be modified hash-max-ziplist-value option and the hash-max-ziplist-entries in Reids profile option.

A collection of objects (set)

Coded collection of objects can be intset and one hashtable.

(1) intset encoding

intset collection of objects underlying encoded set of integers is achieved, all the elements are stored in a set of integers.

(2) hashtable encoding

hashtable collection object underlying implementation is encoded dictionary, the dictionary of each key is a string object, to save a collection element, except that the values of the dictionary are NULL; refer hashset structure of java.

Transcoding collection of objects:

• Intset encoded using a collection of objects need to meet two conditions: First, all elements are integer values; the second is the number of elements less than or equal 512; does not satisfy any one will use hashtable coding.
• More second condition may be modified et-max-intset-entries option Redis configuration file.

Ordered set of objects (zset)

Ordered set of encoding can be ziplist and one skiplist.

(1) ziplist encoding

An ordered set of objects is achieved the underlying ziplist compression coding list, which is similar to the structure of the object hash, except that the compression two closely linked list of nodes, the first member holding the elements, the second elements stored value, and small value close to the header, footer and close large.

(2) skiplist encoding

an ordered set of objects is achieved the underlying skiplist encoded two kinds of jumping and dictionaries;
each node keeps a table hop collection element, and press the value in ascending order; object attribute storage node of the member elements, score attribute holds score;
dictionary of each key to save a collection element, members of the key elements to save the dictionary, and the dictionary values are saved scores.

Why skiplist encoding to use both jumping and dictionaries to achieve?

• Jump Table advantage is ordered, but the score of the query complexity O (logn); dictionary query score complexity O (1), but the disorder, so even a little be implemented in conjunction with structure.
• Although the use of elemental members and two scores, but the structure of the collection is shared by the two structures pointer to the same address, without wasting memory.

Ordered set of encoding conversion:

• Encoding the ordered collection object uses ziplist need to meet two conditions: First, all the elements of a length less than 64 bytes; the second is the number of elements less than 128; a do not satisfy any of the conditions used skiplist coding.
• The above two conditions can be modified zset-max-ziplist-entries and options zset-max-ziplist-value option Redis configuration file.

Scenarios

redis general scenarios

• Cache session (single sign-on)
• Distributed Lock, such as: the use of setnx
• Or lists of the best counter
• Product List, or user data base lists, etc.
• Columns using the list as the message
• Spike, inventory and other deductions

Five types of application scenarios

• String, redis KV for high efficiency operation can be directly used as a counter. For example, the number of online statistics, etc., the other type is a string of binary storage security, so you can use it to store images and even video.
• hash, stored key-value pair may be generally used to keep the basic attribute information of an object, e.g., user information, product information, etc. In addition, since the size of the hash used is less than the configured size when the structure is ziplist, more cost memory, so for a large number of data storage may be considered to use the hash to achieve the amount of fragmentation of data compression, memory saving purposes, e.g., for the name of the picture address corresponding to the large quantities of goods. For example: a fixed commodity code is 10 bits, the former may be selected as a key hash 7 after a three Field, picture as an address value. Each hash table so that no more than 999, as long as the hash-max-ziplist-entries redis.conf was changed to 1024, can.
• list, list type, can be used to implement the message queue, you can also use it to provide a range command, do pagination queries.
• set, collection, ordered list of integers can be used directly set. Some weight functions may be used to, for example, a user name can not be repeated, etc. In addition, also can be set intersection, and set operations, certain elements common to find
• zset, ordered set, can be used to find the range, leaderboards or topN function.

to sum up

In five of the data object Redis, string object is the only one of the other four can be embedded as a target data object;

List (List), hash (the hash), set (SET), an ordered set (zset) underlying implementation uses both compressed condition list structure, a list structure using a compression and are relatively small in number of elements, the byte length case of short;

The advantage of using four data compression list of objects:
(1) save memory, reducing memory overhead, the Redis memory database is, therefore reducing memory overhead is necessary under certain circumstances.
(2) reduce memory fragmentation, compression memory block list is continuous, and the number of allocated memory once.
(3) compressing the new list, delete, search operation the average time complexity is O (N), N in a certain range again, this time is almost negligible, and the upper limit value of N is configurable .
(4) four data object has two coding structure, increased flexibility.