Redis queue and Stream
The biggest new feature of Redis 5.0 is the addition of a data structure Stream, which is a new powerful and durable message queue that supports multicast. The author declares that Redis Stream borrows from Kafka's design.
The structure of Redis Stream is shown in the figure above. Each Stream has a message list, which strings all the added messages together. Each message has a unique ID and corresponding content. The message is persistent, and the content is still there after Redis restarts.
Each Stream has a unique name, which is the key of Redis, which is automatically created when we first use the xadd command to append a message.
Each Stream can be linked to multiple consumer groups, and each consumer group will have a cursor last_delivered_id that moves forward on the Stream array, indicating which message the current consumer group has consumed. Each consumer group has a unique name in the Stream. The consumer group will not be created automatically. It needs to be created with a separate command xgroup create. It needs to specify a certain message ID of the Stream to start consumption. This ID is used to initialize the last_delivered_id variable.
The state of each consumer group (Consumer Group) is independent and does not affect each other. That is to say, the messages inside the same Stream will be consumed by every consumer group.
The same consumer group (Consumer Group) can be attached to multiple consumers (Consumer), and these consumers are in a competitive relationship. Any consumer who reads the message will move the cursor last_delivered_id forward. Each consumer has a unique name within the group.
There will be a state variable pending_ids inside the consumer (Consumer), which records the messages that have been read by the client but have not yet been acknowledged. If the client does not have ack, the message ID in this variable will increase, and once a message is acked, it will start to decrease. This pending_ids variable is officially called PEL in Redis, that is, Pending Entries List. This is a very core data structure, which is used to ensure that the client consumes the message at least once, and will not be lost in the middle of the network transmission. deal with.
The format of the message ID is timestampInMillis-sequence, for example, 1527846880572-5, which indicates that the current message is generated at the millisecond timestamp 1527846880572, and it is the fifth message generated within this millisecond. The message ID can be automatically generated by the server or specified by the client itself, but the form must be integer-integer, and the ID of the message added later must be greater than the ID of the previous message.
The content of the message is a key-value pair, which is shaped like a hash-structured key-value pair, which is nothing special.
Common Operation Commands
production side
xadd append message
xdel deletes the message, the deletion here is only to set the flag, and will not actually delete the message.
xrange gets the message list and automatically filters the deleted messages
xlen message length
del delete Stream
xadd streamtest * name mark age 18
streamtest indicates the name of the current queue, which is the key in Redis in our general sense. The * sign indicates that the server automatically generates an ID, followed by "name mark age 18", which is the message we store in the current streamtest queue. It is also the storage form of key/value
The return value 1626705954593-0 is the generated message ID, which consists of two parts: timestamp-serial number. The timestamp is in milliseconds and is the time of the Redis server that generated the message. It is a 64-bit integer. The sequence number is the sequence number of the message at this millisecond time point. It is also a 64-bit integer.
In order to ensure that the messages are in order, the IDs generated by Redis are monotonically increasing in order. Since the ID contains a timestamp part, in order to avoid problems caused by server time errors (for example, the server time is delayed), each Stream type data of Redis maintains a latest_generated_id attribute, which is used to record the ID of the last message. If it is found that the current timestamp is backward (less than the record of latest_generated_id), then use the scheme that the timestamp remains unchanged and the serial number is incremented as the new message ID (this is why the serial number uses int64 to ensure that there are enough serial numbers), Thus, the monotonically increasing property of ID is guaranteed.
If there are no very special needs, it is strongly recommended to use the Redis scheme to generate message IDs, because this monotonically increasing ID scheme of timestamp + serial number can meet almost all requirements, but the ID supports customization.
xrange streamtest - +
Among them - represents the minimum value, + represents the maximum value
Or we can specify a list of message IDs:
xdel streamtest 1626706380924-0
xlen streamtest
del streamtest deletes the entire Stream
Consumer
single consumer
Although there is a concept of consumer groups in Stream, it is possible to independently consume Stream messages without defining a consumer group, and even block and wait when there are no new messages in Stream. Redis has designed a separate consumption instruction xread, which can use Stream as an ordinary message queue (list). When using xread, we can completely ignore the existence of Consumer Group, just like Stream is an ordinary list.
xread count 1 streams stream2 0-0
"count 1" means to read 1 message from Stream, the default is of course the header, "streams" can be understood as a Redis keyword, "stream2" indicates the name of the queue to be read, and "0-0" means starting from the beginning
xread count 2 streams stream2 1626710882927-0
You can also specify to start from the message Id of streams (excluding the message id in the command)
xread count 1 streams stream2 $
$ stands for reading from the end, the above means to read the latest message from the end, and no message is returned by default at this time
So it is best to read the latest message at the tail in a blocking manner until a new message arrives
xread block 0 count 1 streams stream2 $
The number after the block represents the blocking time in milliseconds
At this point we open a new client and write a message to stream2
It can be seen that the blocking is lifted, the new message content is returned, and a waiting time is also displayed, here we waited for 127.87s
Generally speaking, if the client wants to use xread for sequential consumption, it must remember where the current consumption is, that is, the returned message ID. Next time you continue to call xread, pass the last message ID returned last time as a parameter, and you can continue to consume subsequent messages.
consumer group
Create a consumer group
Stream creates a consumer group (Consumer Group) through the xgroup create command, and needs to pass the initial message ID parameter to initialize the last_delivered_id variable.
xgroup create stream2 cg1 0-0
"stream2" indicates the name of the queue to be read, "cg1" indicates the name of the consumer group, and "0-0" indicates consumption from the beginning
xgroup create stream2 cg2 $
$ means to start consumption from the end, only new messages are accepted, and all current Stream messages will be ignored
Now we can use the xinfo command to see the situation of stream2:
xinfo stream stream2
xinfo groups stream2
message consumption
With a consumer group, consumers are naturally needed. Stream provides the xreadgroup command to consume within a consumer group. It is necessary to provide the consumer group name, consumer name, and initial message ID.
Like xread, it can also block waiting for new messages. After reading a new message, the corresponding message ID will enter the consumer's PEL (message being processed) structure, and the client will use the xack command to notify the server after processing that the message has been processed, and the message ID will be transferred from the PEL removed from the .
xreadgroup GROUP cg1 c1 count 1 streams stream2 >
"GROUP" is a keyword, "cg1" is the name of the consumption group, "c1" is the name of the consumer, "count 1" indicates the consumption quantity, and the > sign means to start reading from the last_delivered_id of the current consumption group. Whenever a consumer reads Fetch a message and the last_delivered_id variable will advance
When we defined cg1 earlier, we consumed it from scratch, so we naturally got the first message in Stream2
Execute the above command again
Naturally, the next message was read.
We finish reading the message in Stream2
xreadgroup GROUP cg1 c1 count 2 streams stream2 >
Naturally there is no message to read, xreadgroup GROUP cg1 c1 count 1 streams stream2 >
Then set the blocking wait
xreadgroup GROUP cg1 c1 block 0 count 1 streams stream2 >
We open a new client and send a message to stream2
xadd stream2 * name lison score 98
Return to the original client, find that the blocking is lifted, and a new message is received
Let's observe the status of the consumer group
If there are multiple consumers in the same consumer group, we can also observe the status of each consumer through the xinfo consumers command
xinfo consumers stream2 cg1
It can be seen that the consumer c1 currently has 5 messages waiting for ACK, and has not read any messages after being idle for 441340 ms.
If we confirm a message
xack stream2 cg1 1626751586744-0
You can see that the message to be confirmed has become 4
xack allows multiple message ids, such as
At the same time, Stream also provides the command XPENDIING to obtain the unprocessed messages of the consumer group or consumers within the consumer. Each Pending message has 4 attributes:
message ID
Belonging to consumers
IDLE, read time
delivery counter, the number of times the message has been read
Command XCLAIM is used to perform message transfer operations, and transfer a message to its own Pending list. It is necessary to set the group, the target consumer of the transfer, and the message ID. At the same time, it is necessary to provide IDLE (the length of time it has been read). Only after this length of time can it be transferred.
For more Redis Stream commands, please refer to the official Redis documentation:
https://redis.io/topics/streams-intro
At the same time, in the Redis document, "Related commands" will be displayed on the right side of the details page of each command. Through this list, you can quickly learn about related commands and enter the details page of specific commands.