Redis from entry to abandonment series (3) List

Redis from entry to abandonment series (3) List

The example in this article is based on: 5.0.4 List is a relatively common data structure in Redis, which is implemented as quicklist, which is a doubly linked list of ziplists

Redis from entry to abandonment series (1) String

Redis from entry to abandonment series (2) Hash

First let's see how to use the List type in redis

//设置key的列表为value
lpush key value [value...]

Code example:

//栈的用法，rpush rpop一样~ 通过rpush,lpop相当于堆的用法
> lpush books java python c
(integer) 3
> lpop books
"c"
> lpop books
"python"
> lpop books
"java"
----------------------------------
//返回列表key指定区间的元素,区间偏移量start跟stop指定
//start跟stop的下表都是以0为底
> lrange books 0 2
1) "c"
2) "python"
3) "java"
----------------------------------
//ltrim可以作为一个定长的list，每次都可以获取到最新的2条数据
> lpush books java python c c++
(integer) 4
> ltrim books 0 1
OK
> lrange books 0 -1
1) "c++"
2) "c"
----------------------------------
//当给定列表内没有任何元素可供弹出的时候，连接将被blpop ,brpop命令阻塞，直到等待超时或发现可弹出元素为止。
//设置超时 1秒
> BLPOP books 1
1) "books"
2) "c++"
> BLPOP books 1
1) "books"
2) "c"
> BLPOP books 1
(nil)
(1.05s)
----------------------------------

So far, the usage of redis list has come to an end.

---

Source code analysis

At the beginning of this article, the list is implemented as quicklist, and quicklist is a doubly linked list of ziplist, so what is its internal structure?

/* quicklist is a 40 byte struct (on 64-bit systems) describing a quicklist.
 * 'count' is the number of total entries.
 * 'len' is the number of quicklist nodes.
 * 'compress' is: -1 if compression disabled, otherwise it's the number
 *                of quicklistNodes to leave uncompressed at ends of quicklist.
 * 'fill' is the user-requested (or default) fill factor. */
typedef struct quicklist {
    quicklistNode *head;
    quicklistNode *tail;
    unsigned long count;        /* total count of all entries in all ziplists */
    unsigned long len;          /* number of quicklistNodes */
    int fill : 16;              /* fill factor for individual nodes */
    unsigned int compress : 16; /* depth of end nodes not to compress;0=off */
} quicklist;

/* quicklistNode is a 32 byte struct describing a ziplist for a quicklist.
 * We use bit fields keep the quicklistNode at 32 bytes.
 * count: 16 bits, max 65536 (max zl bytes is 65k, so max count actually < 32k).
 * encoding: 2 bits, RAW=1, LZF=2.
 * container: 2 bits, NONE=1, ZIPLIST=2.
 * recompress: 1 bit, bool, true if node is temporarry decompressed for usage.
 * attempted_compress: 1 bit, boolean, used for verifying during testing.
 * extra: 10 bits, free for future use; pads out the remainder of 32 bits */
typedef struct quicklistNode {
    struct quicklistNode *prev;
    struct quicklistNode *next;
    unsigned char *zl;
    unsigned int sz;             /* ziplist size in bytes */
    unsigned int count : 16;     /* count of items in ziplist */
    unsigned int encoding : 2;   /* RAW==1 or LZF==2 */
    unsigned int container : 2;  /* NONE==1 or ZIPLIST==2 */
    unsigned int recompress : 1; /* was this node previous compressed? */
    unsigned int attempted_compress : 1; /* node can't compress; too small */
    unsigned int extra : 10; /* more bits to steal for future usage */
} quicklistNode;
/* quicklistLZF is a 4+N byte struct holding 'sz' followed by 'compressed'.
 * 'sz' is byte length of 'compressed' field.
 * 'compressed' is LZF data with total (compressed) length 'sz'
 * NOTE: uncompressed length is stored in quicklistNode->sz.
 * When quicklistNode->zl is compressed, node->zl points to a quicklistLZF */
typedef struct quicklistLZF {
    unsigned int sz; /* LZF size in bytes*/
    char compressed[];
} quicklistLZF;

From the above we can know that quicklist is a doubly linked list, so when we use lpush, rpop and other operations, it is O(1).

The ziplist itself is also a list that can maintain the order of data (according to the insertion position), and is a memory-compact list. When we want to indicate that the list has 12 data items, then there may be a variety of choices, such as a quicklist of 3 nodes, and each node ziplist contains 4 data items. Or a quicklist of 2 nodes, each The node ziplist contains 6 data items, so how does redis choose? We can find clues in redis.conf~

# Lists are also encoded in a special way to save a lot of space.
# The number of entries allowed per internal list node can be specified
# as a fixed maximum size or a maximum number of elements.
# For a fixed maximum size, use -5 through -1, meaning:
# -5: max size: 64 Kb  <-- not recommended for normal workloads
# -4: max size: 32 Kb  <-- not recommended
# -3: max size: 16 Kb  <-- probably not recommended
# -2: max size: 8 Kb   <-- good
# -1: max size: 4 Kb   <-- good
# Positive numbers mean store up to _exactly_ that number of elements
# per list node.
# The highest performing option is usually -2 (8 Kb size) or -1 (4 Kb size),
# but if your use case is unique, adjust the settings as necessary.
list-max-ziplist-size -2

# Lists may also be compressed.
# Compress depth is the number of quicklist ziplist nodes from *each* side of
# the list to *exclude* from compression.  The head and tail of the list
# are always uncompressed for fast push/pop operations.  Settings are:
# 0: disable all list compression
# 1: depth 1 means "don't start compressing until after 1 node into the list,
#    going from either the head or tail"
#    So: [head]->node->node->...->node->[tail]
#    [head], [tail] will always be uncompressed; inner nodes will compress.
# 2: [head]->[next]->node->node->...->node->[prev]->[tail]
#    2 here means: don't compress head or head->next or tail->prev or tail,
#    but compress all nodes between them.
# 3: [head]->[next]->[next]->node->node->...->node->[prev]->[prev]->[tail]
# etc.
list-compress-depth 0

list-max-ziplist-size

When set to a positive number, it means that only the number of elements can be stored at most. The author of redis recommends setting it to -1 or -2, setting the size of the elements that the ziplist on each quicklist node can store~ When the list is very long, the middle The frequency of the data being accessed may be very low, then in this case, list provides a parameter to compress the intermediate data~

list-compress-depth 0

This parameter indicates the number of uncompressed nodes at both ends of the quicklist. The head node and tail node are always uncompressed, which is convenient for quick access at both ends of the list

• 0: is a special value that means no compression. This is the default for Redis.
• 1: Indicates that one node at both ends of the quicklist is not compressed, and the nodes in the middle are compressed.
• 2: Indicates that two nodes at both ends of the quicklist are not compressed, and the nodes in the middle are compressed.
• 3: Indicates that 3 nodes at both ends of the quicklist are not compressed, and the nodes in the middle are compressed.
• So on and so forth…

The quickList structure diagram is shown in the following figure:

The configuration size and node compression depth configuration of the corresponding ziplist in the figure are as follows:

list-max-ziplist-size 3
list-compress-depth 1

In this example, we can see that there is a node at both ends of the quickList that is not compressed, and their data pointers point to the real ziplist (that is, the pointer to zl). The other nodes in the middle are compressed, and their data pointers point to quicklistLZF

Application scenarios

1. Message queue (no ack mechanism)

//生产者使用lpush将消息放入list中，消费者就可以通过rpop取出该消息,并且可以保证消息的有序性。
>lpush message "ces"
(integer) 1
>rpop message
"ces"

2. Timeline

//一种场景就是当用户发送一条微博,通过lpush将它存放到list中,然后通过lrange就可以取出最近的最新的微博信息了
> lpush weibo "xiaoxi1"
(integer) 1
> lpush weibo "xiaoxi2"
(integer) 2
> lpush weibo "xiaoxi3"
(integer) 3
> lrange weibo 0 9
1) "xiaoxi3"
2) "xiaoxi2"
3) "xiaoxi1"

{{o.name}}

{{m.name}}