Getting started with Redis, starting from the underlying data structure and application principles

table of Contents

One, know Redis

Two, Redis 5 big data types

Three, Redis SDS underlying data structure analysis

Four, linked list data structure analysis

Five, list data structure

1. Compressed list

2. Quick list

Six, Redis dictionary

Seven, Redis integer collection

Eight, Redis jump table

Nine, Redis object

Ten, Redis usage scenarios

---

One, know Redis

Recently I started to learn Redis database, learn from the simplest content, and then summarize and record it here, in case you forget, and share it with everyone!

Windows installation is shown in the figure:

download link:

Windows：https://github.com/microsoftarchive/redis/releases/download/win-3.2.100/Redis-x64-3.2.100.zip

Linux：http://download.redis.io/releases/redis-6.0.6.tar.gz

Server side:

Client side:

As a high-performance, open-source NoSQL database, Redis mainly stores data in the form of key-value pairs, developed in C language, and adopts the BSD protocol. It has powerful functions and supports multiple data types. It is a non-relational database.

It is different from relational databases such as Mysql, SQLServer, and Oracle. Therefore, the characteristics of Redis give it many advantages:

1. Support multiple programming languages, Such as Java, C, C++, Python, PHP, Go, Lua, etc.;
2. Including rich data types, Such as String, List, Set, Hash, Sorted Set;
3. Fast reading and writing speed and high performance. Official data: read speed 110000 times/s, write speed 81000 times/s, which means that data is stored in memory for reading and writing;
4. Persistence. The important feature of Redis is to achieve persistence, save the data in memory to disk regularly, and load it into memory again when restarting the server. There are AOF and RDB for persistence;
5. Simple and powerful. Can realize news subscription publication, Lua script, database transaction, etc. Redis is a single-threaded work, all operations are atomic and simple to use.
6. Realize high-availability master-slave replication.
7. Realize distributed cluster and high availability, respectively Redis Cluster and Redis Sentinel.
8. Support multiple data structures. Such Hash, Set, bitmap, etc.

The following briefly introduces the five data types of Redis.

Two, Redis 5 big data types

The specific operation commands of Redis 5 big data types can be found at: http://www.redis.cn/commands.html

1. String

Strings are the most basic data type in Redis and are binary safe. The String type key can store up to 512M of data.

Supported data includes: binary data, serialized data, JSONized objects, etc.

2. Hash

The Hash type is a mapping table of String type fields and values, which is commonly used to store object information. Each hash table can store 2^32-1 key-value pairs, which is equivalent to 4 billion pieces of data.

3. List

The Redis list is equivalent to a simple string list, sorted in the order of insertion, and an element can be inserted into the head or tail of the table. The same can store 2^32-1 elements.

4. Set

The Set data type is an unordered collection of String type, and each element is unique. The collection is realized by a hash table, with high efficiency of addition, deletion, and O(1) complexity. The maximum capacity of a collection is to store 2^32-1 elements.

5. Sorted Sort

An ordered collection is a collection of String type, in which each element corresponds to a value of type double. Redis sorts the elements according to the size of this value, which is also implemented through a hash table, with high efficiency of addition, deletion, and O(1) complexity. The maximum capacity of a collection is to store 2^32-1 elements.

Three, Redis SDS underlying data structure analysis

Redis is written in C language, and the underlying implementation has the grammar and characteristics of C language. The string type in C is not used directly, but the SDS (simple dynamic string) type is built by itself as the default string of Redis.

SDS part of the source code:

struct __attribute__ ((__packed__)) sdshdr5 {
    unsigned char flags; /* 3 lsb of type, and 5 msb of string length */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr8 {
    uint8_t len; /* used */
    uint8_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr16 {
    uint16_t len; /* used */
    uint16_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr32 {
    uint32_t len; /* used */
    uint32_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr64 {
    uint64_t len; /* used */
    uint64_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};

It can be seen from the underlying code that a variety of structures are defined to adapt to different needs.

The difference between C string and SDS:

1. C's string API is binary insecure. There may be a buffer overflow. Only text data can be saved. All library functions of string.h can be used. Once the length is modified, the memory is allocated once. The time complexity of obtaining the string length is O( n).
2. SDS API is binary safe, there is no buffer overflow, you can save text/binary data, use some library functions of string.h, modify the length of N times and allocate memory times <=N, get the length to O(1).

Four, linked list data structure analysis

/* Node, List, and Iterator are the only data structures used currently. */

typedef struct listNode {
    struct listNode *prev;
    struct listNode *next;
    void *value;
} listNode;

typedef struct listIter {
    listNode *next;
    int direction;
} listIter;

typedef struct list {
    listNode *head;
    listNode *tail;
    void *(*dup)(void *ptr);
    void (*free)(void *ptr);
    int (*match)(void *ptr, void *key);
    PORT_ULONG len;
} list;

 There is no built-in linked list structure in the C language. Redis builds a linked list by itself. It uses a doubly linked list. It consists of multiple discretely distributed nodes connected by pointers. Each node has a predecessor node and a successor node, except for the head node ( No predecessor) and tail node (no successor).

Five, list data structure

1. Compressed list

#ifndef _ZIPLIST_H
#define _ZIPLIST_H

#define ZIPLIST_HEAD 0
#define ZIPLIST_TAIL 1

unsigned char *ziplistNew(void);
unsigned char *ziplistMerge(unsigned char **first, unsigned char **second);
unsigned char *ziplistPush(unsigned char *zl, unsigned char *s, unsigned int slen, int where);
unsigned char *ziplistIndex(unsigned char *zl, int index);
unsigned char *ziplistNext(unsigned char *zl, unsigned char *p);
unsigned char *ziplistPrev(unsigned char *zl, unsigned char *p);
unsigned int ziplistGet(unsigned char *p, unsigned char **sval, unsigned int *slen, PORT_LONGLONG *lval);
unsigned char *ziplistInsert(unsigned char *zl, unsigned char *p, unsigned char *s, unsigned int slen);
unsigned char *ziplistDelete(unsigned char *zl, unsigned char **p);
unsigned char *ziplistDeleteRange(unsigned char *zl, int index, unsigned int num);
unsigned int ziplistCompare(unsigned char *p, unsigned char *s, unsigned int slen);
unsigned char *ziplistFind(unsigned char *p, unsigned char *vstr, unsigned int vlen, unsigned int skip);
unsigned int ziplistLen(unsigned char *zl);
size_t ziplistBlobLen(unsigned char *zl);

#ifdef REDIS_TEST
int ziplistTest(int argc, char *argv[]);
#endif

#endif /* _ZIPLIST_H */

 Redis compressed list is implemented by the bottom layer of list key and hash key.

When the list contains fewer elements and the value is smaller, Redis uses a compressed list to implement it, which is a sequential data structure.

2. Quick list

The quick list is a doubly linked list composed of compressed lists, and each node of the linked list saves data in a compressed list.

The list is defined as follows:

typedef struct quicklist {
    quicklistNode *head;
    quicklistNode *tail;
    PORT_ULONG count;        /* total count of all entries in all ziplists */
    unsigned int len;           /* number of quicklistNodes */
    int fill : 16;              /* fill factor for individual nodes */
    unsigned int compress : 16; /* depth of end nodes not to compress;0=off */
} quicklist;

Quick list node:

typedef struct quicklistNode {
    struct quicklistNode *prev;
    struct quicklistNode *next;
    unsigned char *zl;
    unsigned int sz;             /* ziplist size in bytes */
    unsigned int count : 16;     /* count of items in ziplist */
    unsigned int encoding : 2;   /* RAW==1 or LZF==2 */
    unsigned int container : 2;  /* NONE==1 or ZIPLIST==2 */
    unsigned int recompress : 1; /* was this node previous compressed? */
    unsigned int attempted_compress : 1; /* node can't compress; too small */
    unsigned int extra : 10; /* more bits to steal for future usage */
} quicklistNode;

 

Six, Redis dictionary

The Redis dictionary is a data structure used to store Redis key-value pairs. It is also constructed by Redis itself. The bottom layer of the Redis database is also implemented by a dictionary, and the CURD operation is built on the dictionary.

typedef struct dictEntry {
    void *key;
    union {
        void *val;
        uint64_t u64;
        int64_t s64;
        double d;
    } v;
    struct dictEntry *next;
} dictEntry;

typedef struct dictType {
    unsigned int (*hashFunction)(const void *key);
    void *(*keyDup)(void *privdata, const void *key);
    void *(*valDup)(void *privdata, const void *obj);
    int (*keyCompare)(void *privdata, const void *key1, const void *key2);
    void (*keyDestructor)(void *privdata, void *key);
    void (*valDestructor)(void *privdata, void *obj);
} dictType;

/* This is our hash table structure. Every dictionary has two of this as we
 * implement incremental rehashing, for the old to the new table. */
typedef struct dictht {
    dictEntry **table;
    PORT_ULONG size;
    PORT_ULONG sizemask;
    PORT_ULONG used;
} dictht;

typedef struct dict {
    dictType *type;
    void *privdata;
    dictht ht[2];
    PORT_LONG rehashidx; /* rehashing not in progress if rehashidx == -1 */
    int iterators; /* number of iterators currently running */
} dict;

 

Seven, Redis integer collection

When a collection contains only integer-valued elements and the number of elements is small, Redis uses this data structure to implement the bottom layer of the collection key.

#ifndef __INTSET_H
#define __INTSET_H
#include <stdint.h>

typedef struct intset {
    uint32_t encoding;
    uint32_t length;
    int8_t contents[];
} intset;

intset *intsetNew(void);
intset *intsetAdd(intset *is, int64_t value, uint8_t *success);
intset *intsetRemove(intset *is, int64_t value, int *success);
uint8_t intsetFind(intset *is, int64_t value);
int64_t intsetRandom(intset *is);
uint8_t intsetGet(intset *is, uint32_t pos, int64_t *value);
uint32_t intsetLen(intset *is);
size_t intsetBlobLen(intset *is);

#ifdef REDIS_TEST
int intsetTest(int argc, char *argv[]);
#endif

#endif // __INTSET_H

Eight, Redis jump table

The jump table is an ordered data structure that supports fast node search and batch processing of nodes. If the ordered set has many elements and the element value is a long string, Redis uses a jump table as the underlying implementation of the ordered set.

Nine, Redis object

The above data structure of Redis is not actually used directly, but an object system is created, including string objects, list objects, and so on. The Red is object system implements a memory recycling mechanism based on reference counting technology. When an object is not in use, the system will automatically reclaim the memory space occupied by the object.

robj *dupStringObject(robj *o) {
    robj *d;

    serverAssert(o->type == OBJ_STRING);

    switch(o->encoding) {
    case OBJ_ENCODING_RAW:
        return createRawStringObject(o->ptr,sdslen(o->ptr));
    case OBJ_ENCODING_EMBSTR:
        return createEmbeddedStringObject(o->ptr,sdslen(o->ptr));
    case OBJ_ENCODING_INT:
        d = createObject(OBJ_STRING, NULL);
        d->encoding = OBJ_ENCODING_INT;
        d->ptr = o->ptr;
        return d;
    default:
        serverPanic("Wrong encoding.");
        break;
    }
}

robj *createQuicklistObject(void) {
    quicklist *l = quicklistCreate();
    robj *o = createObject(OBJ_LIST,l);
    o->encoding = OBJ_ENCODING_QUICKLIST;
    return o;
}

robj *createZiplistObject(void) {
    unsigned char *zl = ziplistNew();
    robj *o = createObject(OBJ_LIST,zl);
    o->encoding = OBJ_ENCODING_ZIPLIST;
    return o;
}

robj *createSetObject(void) {
    dict *d = dictCreate(&setDictType,NULL);
    robj *o = createObject(OBJ_SET,d);
    o->encoding = OBJ_ENCODING_HT;
    return o;
}

robj *createIntsetObject(void) {
    intset *is = intsetNew();
    robj *o = createObject(OBJ_SET,is);
    o->encoding = OBJ_ENCODING_INTSET;
    return o;
}

robj *createHashObject(void) {
    unsigned char *zl = ziplistNew();
    robj *o = createObject(OBJ_HASH, zl);
    o->encoding = OBJ_ENCODING_ZIPLIST;
    return o;
}

robj *createZsetObject(void) {
    zset *zs = zmalloc(sizeof(*zs));
    robj *o;

    zs->dict = dictCreate(&zsetDictType,NULL);
    zs->zsl = zslCreate();
    o = createObject(OBJ_ZSET,zs);
    o->encoding = OBJ_ENCODING_SKIPLIST;
    return o;
}

robj *createZsetZiplistObject(void) {
    unsigned char *zl = ziplistNew();
    robj *o = createObject(OBJ_ZSET,zl);
    o->encoding = OBJ_ENCODING_ZIPLIST;
    return o;
}

Ten, Redis usage scenarios

After talking about so many Redis theoretical knowledge points, how can the advantages be seen?

In practical applications, Redis can be introduced into major websites and systems, and it is widely used, which solves the problems that relational databases cannot solve.

Redis usage scenarios:

1. Do caching. The most common usage scenario of Redis. It does not need to regenerate data every time, and the cache speed and query speed are fast. Cache news content, product information, or shopping carts, etc.
2. Do a counter. Redis operations are atomic. Can be used to record the number of reposts and comments.
3. Realize the message queue system. Support pattern matching, which can realize news subscription and publishing. Blocking queue commands can realize activities such as spikes and panic buying.
4. Real-time system, message system. The Redis set function can be used to view the user's specific operations and realize a real-time system.
5. Billion-level rankings. This is also an important application of Redis, which uses ordered collections to rank hundreds of millions of users in real time, thanks to the read and write speed of Redis.
6. Large social network. Such as QQ, Weibo, etc., users need Redis support for browsing and chatting.

This article is Xiaobai's recent introduction to learning Redis database, learning records from the simplest content, and sharing it with everyone! As a high-performance, open source NoSQL database, Redis is different from relational databases such as Mysql, SQLServer, and Oracle. Therefore, the characteristics of Redis give it many advantages. There are many important places that are not recorded here, such as the specific implementation principles of persistence AOF, RDB, Redis cluster, etc., after learning later, I will be free to record!

If you think it's good, welcome to "one-click, three-link", like, bookmark, follow, comment directly if you have any questions, exchange and learn! 

---

My CSDN blog: https://blog.csdn.net/Charzous/article/details/114546348