About the new features of Redis 5.0: Stream data structure

Reprinted from: https://blog.csdn.net/enmotech/article/details/81230531

Make minor modifications

thank!

 

Summary

What are the special functions of Redis Stream? What are the similarities and differences with kafka? How to use it better? The author of this article has a lot of research on this, and after reading it, I feel that it has benefited a lot. You may wish to learn more about it.

 

Content overview

Redis5.0 was recently released by the author suddenly, adding many new features. The biggest new feature of Redis5.0 is the addition of a data structure Stream , which is a new powerful and durable message queue that supports multicast . The author admits that Redis Stream draws heavily on Kafka's design.

The structure of Redis Stream is shown in the figure above. It has a linked list of messages , and all the joined messages are concatenated. Each message has a unique ID and corresponding content. The message is persistent, and after Redis restarts, the content is still there.

Each Stream has a unique name, which is the Redis key , which is automatically createdxadd when we first use the instruction to append the message .

Each Stream can hang multiple consumer groups , and each consumer group will have a cursorlast_delivered_id moving forward on the Stream array, indicating which message has been consumed by the current consumer group. Each consumer group has a unique name in the Stream. The consumer group will not be created automatically. It needs to be created by a separate instruction. It needs xgroup createto specify that a certain message ID of the Stream will start consumption. This ID is used to initialize last_delivered_idvariables.

The status of each consumer group is independent and not affected by each other. In other words, the same internal message of Stream will be consumed by each consumer group.

The same consumer group (Consumer Group) can be connected to multiple consumers (Consumer), these consumers are in a competitive relationship, any consumer reads the message will make the cursor last_delivered_idmove forward . Each consumer has a unique name within the group.

There will be a state variable inside the consumer pending_ids, which records the current message that has been read by the client , but there is no ack yet . If the client does not have an ack, the message ID in this variable will increase. Once a message is acked, it will start to decrease. The pending_ids variable is called Redis official PEL, that is Pending Entries List, it is a very core data structure, which is used to ensure that the client consumes the message at least once , and will not be lost in the middle of the network transmission.

Message ID

The form of the message ID is timestampInMillis-sequence, for example 1527846880572-5, it indicates that the current message 1527846880572is generated at the millimeter time stamp , and is the fifth message generated within the millisecond. The message ID can be automatically generated by the server or specified by the client itself, but the form must be 整数-整数, and the ID of the message added later must be greater than the ID of the previous message.

Message content

The content of the message is a key-value pair. The key-value pair in the form of a hash structure is nothing special.

Add, delete and modify

1. xadd append message

2. xdel deletes the message. The deletion here only sets the flag bit and does not affect the total length of the message

3. xrange gets the message list, it will automatically filter the deleted messages

4. xlen message length

5. del delete Stream

# *号表示服务器自动生成ID，后面顺序跟着一堆key/value
127.0.0.1:6379> xadd codehole * name laoqian age 30  #  名字叫laoqian，年龄30岁
1527849609889-0  # 生成的消息ID
127.0.0.1:6379> xadd codehole * name xiaoyu age 29
1527849629172-0
127.0.0.1:6379> xadd codehole * name xiaoqian age 1
1527849637634-0
127.0.0.1:6379> xlen codehole
(integer) 3
127.0.0.1:6379> xrange codehole - +  # -表示最小值, +表示最大值
127.0.0.1:6379> xrange codehole - +
1) 1) 1527849609889-0
   2) 1) "name"
      2) "laoqian"
      3) "age"
      4) "30"
2) 1) 1527849629172-0
   2) 1) "name"
      2) "xiaoyu"
      3) "age"
      4) "29"
3) 1) 1527849637634-0
   2) 1) "name"
      2) "xiaoqian"
      3) "age"
      4) "1"
127.0.0.1:6379> xrange codehole 1527849629172-0 +  # 指定最小消息ID的列表
1) 1) 1527849629172-0
   2) 1) "name"
      2) "xiaoyu"
      3) "age"
      4) "29"
2) 1) 1527849637634-0
   2) 1) "name"
      2) "xiaoqian"
      3) "age"
      4) "1"
127.0.0.1:6379> xrange codehole - 1527849629172-0  # 指定最大消息ID的列表
1) 1) 1527849609889-0
   2) 1) "name"
      2) "laoqian"
      3) "age"
      4) "30"
2) 1) 1527849629172-0
   2) 1) "name"
      2) "xiaoyu"
      3) "age"
      4) "29"
127.0.0.1:6379> xdel codehole 1527849609889-0
(integer) 1
127.0.0.1:6379> xlen codehole  # 长度不受影响
(integer) 3
127.0.0.1:6379> xrange codehole - +  # 被删除的消息没了
1) 1) 1527849629172-0
   2) 1) "name"
      2) "xiaoyu"
      3) "age"
      4) "29"
2) 1) 1527849637634-0
   2) 1) "name"
      2) "xiaoqian"
      3) "age"
      4) "1"
127.0.0.1:6379> del codehole  # 删除整个Stream
(integer) 1