Interpretation of Redis stream type

A Redis stream (stream) is a data structure that acts like an append-only log, but also implements several operations to overcome some of the limitations of a typical append-only log. These include random access in O(1) time and complex consumption strategies such as consumer groups. You can use streams to record and simultaneously syndicate events in real time.

Redis generates a unique ID for each stream entry. These IDs can be used later to retrieve their associated entries, or to read and process all subsequent entries in the stream.

Redis Stream is mainly used for message queues (MQ, Message Queue) . Redis itself has a Redis publish subscription (pub/sub) to implement the function of message queues, but it has a disadvantage that messages cannot be persisted . If there is a network disconnection , Redis downtime, etc., the message will be discarded. Redis Stream provides message persistence and master-backup replication functions , allowing any client to access data at any time, remembering the access location of each client, and ensuring that messages are not lost. stream has a message linked list, which strings all the added messages together, and each message has a unique ID and corresponding content. The message is persistent, and the content is still there after Redis restarts.

data structure

The structure of Redis Stream is as follows. It has a message list that strings together all the added messages. Each message has a unique ID and corresponding content:

information

Each Stream has a unique name, which is the key of Redis, which is automatically created when we first use the xadd command to append a message.XADD key ID field value[field value ...]

Message ID: The format of the message ID is timestampInMillis-sequence, for example, 1527846880572-5, which means that the current message is generated at the millisecond timestamp 1527846880572, and it is the fifth message generated within this millisecond. The message ID can be automatically generated by the server or specified by the client itself, but the form must be integer-integer, and the ID of the message added later must be greater than the ID of the previous message.
Message content: The message content is a key-value pair, which is shaped like a hash-structured key-value pair.

consumer group

Each Stream can be linked to multiple Consumer Groups, and each Consumer Group will have a cursor last_delivered_id that moves forward on the Stream array, indicating which message the current Consumer Group has consumed. Each consumer group has a unique name in the Stream. The consumer group will not be created automatically. It needs to be created by a separate command. It needs to specify a certain message IDxgroup create of the Stream to start consuming. This ID is used to initialize the last_delivered_id variable. The status of each consumer group is independent and does not affect each other. That is to say, the messages inside the same Stream will be consumed by every consumer group. The same consumer group can be attached to multiple consumers (Consumer), and these consumers are in a competitive relationship. Any consumer who reads the message will move the cursor last_delivered_id forward. Each consumer has a unique name within the group.

consumer

There will be a state variable pending_ids inside the consumer to maintain the unconfirmed id of the consumer. pending_ids records the messages currently read by the client, but there is no ack (Acknowledge character: confirmation character). If the client does not have ack, the message ID in this variable will increase, and once a message is acked, it will start to decrease. This pending_ids variable is officially called PEL in Redis, that is, Pending Entries List. This is a core data structure, which is used to ensure that the client consumes the message at least once, and will not be lost in the middle of the network transmission without being read. deal with.

Basic commands

overview

Message queue related commands:

XADD - add a message to the end
XTRIM - Trim streams to limit length
XDEL - delete message
XLEN - Gets the number of elements the stream contains, i.e. the message length
XRANGE - get message list, will automatically filter deleted messages
XREVRANGE - Get the list of messages in reverse, with IDs from largest to smallest
XREAD - Get a list of messages in a blocking or non-blocking manner

Consumer group related commands:

XGROUP CREATE - Create a consumer group
XREADGROUP GROUP - read messages in a consumer group
XACK - marks a message as "handled"
XGROUP SETID - set a new last delivered message ID for a consumer group
XGROUP DELCONSUMER - delete a consumer
XGROUP DESTROY - Delete a consumer group
XPENDING - display information about pending messages
XCLAIM - Transfer ownership of messages
XINFO - view information about streams and consumer groups;
XINFO GROUPS - print information about consumer groups;
XINFO STREAM - Print stream information

xadd command

The XADD command appends the specified flow entry to the flow of the specified key. If the key does not exist, the key will be automatically created using the stream's entry.

An entry is made up of a set of key-value pairs, which is basically a small dictionary. Key-value pairs are stored in the order given by the user, and commands that read the stream (such as XRANGE or XREAD) are guaranteed to return them in the order they were added via XADD.

XADD key ID field value [field value ...]

key : the name of the queue, create it if it does not exist
ID : message id, we use * to indicate that it is generated by redis, which can be customized, but the increment must be guaranteed by ourselves.
field value : record.

redis> XADD mystream * name Sara surname OConnor
"1601372323627-0"
redis> XADD mystream * field1 value1 field2 value2 field3 value3
"1601372323627-1"
redis> XLEN mystream
(integer) 2
redis> XRANGE mystream - +
1) 1) "1601372323627-0"
   2) 1) "name"
      2) "Sara"
      3) "surname"
      4) "OConnor"
2) 1) "1601372323627-1"
   2) 1) "field1"
      2) "value1"
      3) "field2"
      4) "value2"
      5) "field3"
      6) "value3"

Return Value: The command returns the ID of the added entry. If * is passed for the ID parameter, then the ID is automatically generated, otherwise, the command simply returns the same ID that the user specified during insert.

xtrim command

XTRIM trims the stream to the specified number of items, evicting older items (items with smaller IDs) if necessary.

XTRIM key MAXLEN [~] count

key : queue name
MAXLEN : length
count : number

127.0.0.1:6379> XADD mystream * field1 A field2 B field3 C field4 D
"1601372434568-0"
127.0.0.1:6379> XTRIM mystream MAXLEN 2
(integer) 0
127.0.0.1:6379> XRANGE mystream - +
1) 1) "1601372434568-0"
   2) 1) "field1"
      2) "A"
      3) "field2"
      4) "B"
      5) "field3"
      6) "C"
      7) "field4"
      8) "D"

Return Value: Returns the number of entries removed from the stream.

xdel command

Removes the specified entry from the specified stream and returns the number of successfully removed entries. In the case where the passed ID does not exist, the number returned may not be the same as the passed ID number.

XDEL key ID[ID ...]

key: queue name.
ID: Message ID

> XADD mystream * a 1
1538561698944-0
> XADD mystream * b 2
1538561700640-0
> XADD mystream * c 3
1538561701744-0
> XDEL mystream 1538561700640-0
(integer) 1
127.0.0.1:6379> XRANGE mystream - +
1) 1) 1538561698944-0
   2) 1) "a"
      2) "1"
2) 1) 1538561701744-0
   2) 1) "c"
      2) "3"

Return value: the number of entries removed.

xlen command

Returns the number of entries in the stream. If the specified key does not exist, this command returns 0, as if the stream were empty.

Unlike other Redis types, zero-length streams are possible , so you should call TYPEor EXISTSto check if a key exists. Once there are no entries inside (e.g. XDELafter a call), the stream will not be automatically deleted because there may still be consumer groups associated with it.

redis> XADD mystream * item 1
"1601372563177-0"
redis> XADD mystream * item 2
"1601372563178-0"
redis> XADD mystream * item 3
"1601372563178-1"
redis> XLEN mystream
(integer) 3

Return value: the number of entries contained in the stream

xrange command

The xrange command returns entries in a stream that satisfy a given ID range. Ranges are specified by min and max IDs. All entries with an ID between or equal to one of the two specified IDs (closed range) will be returned.

XRANGE key start end [COUNT count]

key : queue name
start : start value, - means the minimum value
end : the end value, + means the maximum value
count : number

redis> XADD writers * name Virginia surname Woolf
"1601372577811-0"
redis> XADD writers * name Jane surname Austen
"1601372577811-1"
redis> XADD writers * name Toni surname Morrison
"1601372577811-2"
redis> XADD writers * name Agatha surname Christie
"1601372577812-0"
redis> XADD writers * name Ngozi surname Adichie
"1601372577812-1"
redis> XLEN writers
(integer) 5
redis> XRANGE writers - + COUNT 2
1) 1) "1601372577811-0"
   2) 1) "name"
      2) "Virginia"
      3) "surname"
      4) "Woolf"
2) 1) "1601372577811-1"
   2) 1) "name"
      2) "Jane"
      3) "surname"
      4) "Austen"

Return Value: This command returns entries whose IDs match the specified range. The returned entry is complete, meaning the ID and all fields that make up the entry are returned. Also, the entries are returned with their fields and values in XADDexactly the order in which they were added using .

xread command

Read data from one or more streams, returning only entries with IDs greater than the last received ID reported by the caller. This command has a blocking option for waiting for an item to become available, something like BRPOPor BZPOPMINetc.

XREAD[COUNT count][BLOCK milliseconds] STREAMS key[key ...]id[id ...]

count: quantity.
milliseconds: Optional, the number of milliseconds to block, if not set, it is non-blocking mode.
key : the queue name.
id: Message ID.

redis> XREAD COUNT 2 STREAMS mystream writers 0-0 0-0
1) 1) "mystream"
   2) 1) 1) 1526984818136-0
         2) 1) "duration"
            2) "1532"
            3) "event-id"
            4) "5"
            5) "user-id"
            6) "7782813"
      2) 1) 1526999352406-0
         2) 1) "duration"
            2) "812"
            3) "event-id"
            4) "9"
            5) "user-id"
            6) "388234"
2) 1) "writers"
   2) 1) 1) 1526985676425-0
         2) 1) "name"
            2) "Virginia"
            3) "surname"
            4) "Woolf"
      2) 1) 1526985685298-0
         2) 1) "name"
            2) "Jane"
            3) "surname"
            4) "Austen"

Return Value: The command returns an array of results: each element of the returned array is an array of two elements (the key name and the entry reported for that key). The reported entry is the full flow entry, with an ID and a list of all fields and values. The entries are returned with their fields and values in XADDexactly the order in which they were added using .

When using BLOCK , an empty reply ( ) will be returned on timeout nil.

xgroup command

Use XGROUP CREATE to create a consumer group, the syntax format is:

XGROUP [CREATE key groupname id-or-$] [SETID key groupname id-or-$] [DESTROY key groupname] [DELCONSUMER key groupname consumername]

key : the name of the queue, create it if it does not exist
groupname : group name.
$ : Indicates consumption from the end, only new messages are accepted, and all current Stream messages are ignored.

Consume from scratch:

XGROUP CREATE mystream consumer-group-name 0-0

Consume from the end:

XGROUP CREATE mystream consumer-group-name $

xreadgroup Command

Use XREADGROUP GROUP to read messages in the consumer group, the syntax format is:

XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] ID [ID ...]

group : consumer group name
consumer : consumer name.
count : The number of reads.
milliseconds : The number of milliseconds to block.
key : the queue name.
ID : Message ID.

xack command

XACKcommand is used to remove one or more messages from the pending entry list (PEL for short) of a stream's consumer group . When a message is delivered to a consumer, it is stored in the PEL for processing, usually as a XREADGROUPside effect of invoking a command, or XCLAIMwhen a consumer takes over the message by invoking a command.

It should be called once a consumer has successfully processed a message, XACKso that the message is not processed again, and as a side effect, the PEL entry for this message is also cleared, freeing memory from the Redis server.

XACK key group ID[ID ...]

Return Value: The command returns the number of successfully acknowledged messages. Certain message IDs may no longer be part of the PEL (for example because they have been acknowledged), and XACKthey will not be counted towards the number of successful acknowledgments.