table of Contents
Redis Stream is a newly added data structure in Redis 5.0 version.
Redis Stream is mainly used for message queues (MQ, Message Queue). Redis itself has a Redis publish and subscribe (pub/sub) to implement the message queue function, but it has the disadvantage that the message cannot be persisted. If there is a network disconnect , Redis downtime, etc., the message will be discarded.
Simply put, publish and subscribe (pub/sub) can distribute messages, but cannot record historical messages.
Redis Stream provides message persistence and master-backup replication functions, allowing any client to access data at any time, and remember the location of each client's access, and ensure that the message is not lost.
The structure of Redis Stream is as follows. It has a message linked list, which links all the added messages together. Each message has a unique ID and corresponding content:
Each Stream has a unique name, which is the Redis key, which is automatically created when we use the xadd command to append a message for the first time.
Analysis of the above picture:
- Consumer Group : Consumer group, created using the XGROUP CREATE command, a consumer group has multiple consumers (Consumers).
- last_delivered_id : cursor, each consumer group will have a cursor last_delivered_id, any consumer reading the message will make the cursor last_delivered_id move forward.
- pending_ids : state variables of consumers (Consumer), which are used to maintain unconfirmed ids of consumers. pending_ids records the messages currently read by the client, but there is no ack (Acknowledge character).
XADD
Use XADD to add messages to the queue. If the specified queue does not exist, create a queue. XADD syntax format:
XADD key ID field value [field value ...]- key : the name of the queue, create it if it does not exist
- ID : The message id, we use * to indicate that it is generated by redis and can be customized, but it is necessary to ensure that it is incremental.
- field value : record.
Xdel and xlength will not be introduced much, let's introduce xtrim.
XTRIM
xtrim trims the stream by expelling older entries (entries with lower IDs).
The stream can be pruned using one of the following strategies:
MAXLEN: As long as the length of the stream exceeds the specified valuethreshold, thethresholdentry will be ejected, which is a positive integer.MINID: Evict entriesthresholdwith lower IDs , wherethresholdis the flow ID.
For example, this will trim the stream to exactly the latest 1000 items:
XTRIM mystream MAXLEN 1000
In this example, all entries with IDs lower than 649085820-0 will be evicted:
XTRIM mystream MINID 649085820
By default, or when optional parameters are provided, the command will perform precise trimming.
According to the strategy, precise trimming means:
MAXLEN: The length of the trimmed stream will be exactly the minimum between its original length and the specified lengththreshold.MINID: The earliest ID in the stream is exactly the smallest value between its original earliest ID and the specified valuethreshold.
XREAD
Use XREAD to obtain the message list in a blocking or non-blocking manner, the syntax format:
XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]- count : number
- milliseconds : optional, the number of milliseconds to block, if not set, it is non-blocking
- key : the name of the queue
- id : message ID
Sometimes when blocking, we only want to receive entries added to the stream via xadd from the moment of blocking. In this case, we are not interested in the history of the added items. In this case, we will use $
Above we go to fetch the data added to the stream from the moment of blocking.
XGROUP
Use this command to manage the consumer groups associated with the stream data structure, you can:
- Create a new consumer group associated with the stream.
- Eliminate a consumer group.
- Remove a specific consumer from the consumer group.
- Set the last delivered ID of the consumer group to another name.
To create a new consumer group, use the following form:
XGROUP CREATE mystream consumer-group-name $
The last parameter is the ID of the last item to be considered for delivery in the stream. In the above case, we use the special ID'$' (which means: the ID of the last item in the stream). In this case, consumers who get data from that consumer group will only see new elements arrive in the stream.
Conversely, if you want the consumer to get the entire flow history, use zero as the starting ID of the consumer group:
XGROUP CREATE mystream consumer-group-name 0
Of course, any other valid ID can also be used. If the specified user group already exists, the command will return -BUSYGROUPan error. Otherwise, perform the operation and return OK. There is no hard limit on the number of consumer groups that can be associated with a given stream.
If the specified stream does not exist when the group is created, an error will be returned. You can use the optional MKSTREAMsubcommand as the last parameter IDto automatically create a stream if it does not exist. Note that if the stream is created this way, its length will be 0:
XGROUP CREATE mystream consumer-group-name $ MKSTREAM
A consumer group can be completely destroyed using the following form:
XGROUP DESTROY mystream consumer-group-name
Even if there are active consumers and pending messages, the consumer group will be destroyed, so make sure to call this command only when you really need it.
Whenever a command mentions a new consumer name, the consumer in the consumer group is automatically created. They can also be created explicitly using the following form:
XGROUP CREATECONSUMER mystream consumer-group-name myconsumer123
To remove a given consumer from the consumer group, the following form can be used:
XGROUP DELCONSUMER mystream consumer-group-name myconsumer123
Sometimes it may be useful to delete old consumers because they are no longer used. This form returns the number of pending messages that the user has before being deleted.
Finally, you can use SETIDsubcommands to set the next message to be delivered. Usually, the next ID is set as the last parameter when creating a user XGROUP CREATE. However, you can modify the next ID later using this form without having to delete and create the user group again. For example, if you want consumers in the consumer group to reprocess all messages in the stream, you can set their next ID to 0:
XGROUP SETID mystream consumer-group-name 0XREADGROUP GROUP
Use XREADGROUP GROUP to read the messages in the consumer group, the syntax format:
XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] ID [ID ...]
- group : consumer group name
- consumer : Consumer name.
- count : the number of reads.
- milliseconds : the number of milliseconds to block.
- key : The name of the queue.
- ID : Message ID.
XREADGROUP GROUP consumer-group-name consumer-name COUNT 1 STREAMS mystream >
