Redis5 message queue for new features Streams

　　Preface
　　
　　5 Redis the new features introduced Streams data structure, it may be present in the characteristic maximum iteration. It makes this version 5.x iteration, when Redis is used as message queues, get better, more powerful native support, which is particularly evident in the persistent message queue. At the same time, stream draws consumer group model concept and design kafka, so that the consumer message processing more efficient and fast. This article analyzes the data structure used Streams API.
　　
　　Ready
　　
　　used herein Redis version 5.0.5. If you are using an earlier 5.x versions, some API results, and described in this article is slightly different.
　　
　　Add a message
　　
　　Streams XADD instruction using the added data is added, operating in a data message in the form of key-value pairs KV. A plurality of key-value pairs may be present message, add Syntax:
　　
　　XADD key ID String Field [... String Field]
　　
　　wherein Streams key name, ID, unique message flag and unrepeatable, field string of key-value pairs to . Here we add the name of the person to flow to operate.
　　
　　XADD person * name ytao des https://ytao.top
　　
　　add the above case, using the ID number * Copy, where the server automatically generated representing Id, return data after addition "1578238486193-0"
　　
　　automatically generated Id herein as < millisecondstime> - <sequencenumber> Id is composed of two parts:
　　
　　millisecondsTime ms time stamp for the current server.
　　
　　sequenceNumber current sequence number, derived from the current value of milliseconds, in order to generate a message, from zero by default incremented by.
　　
　　For example: Rule 4 indicates 1578238486193-3 message time stamp 1578238486193 milliseconds, added.
　　
　　Id In addition to the server automatically generate mode, but also supports the generation of the specified Id, Id but specifies the following conditions:
　　
　　front and rear portions Id must be digital.
　　
　　The minimum Id 0-1, not 0-0, 2-0,3-0 .... but is allowed.
　　
　　Add message, the front half portion Id does not exist smaller than the maximum value Id, Id first half of the second half of the same not smaller latter half maximum ratio exists.
　　
　　Otherwise, when the condition is not satisfied, after the addition will throw an exception:
　　
　　(error) of The ERR ID specified in IS XADD Smaller or equal Within last Top Stream The target Item
　　
　　Indeed, when adding a message, two operations will be performed. The first step, first determine if there is no Streams, Streams name is created, and then add a message to the Streams. Add a message even when, due to the abnormal Id, may also be present in the current name Redis of Streams. Streams of Id can also be used as a pointer because it is an ordered tag.
　　
　　Production, if such use add a message, there will be a problem, that is too large number of messages, make service downtime. Streams here early in the design also has to consider this problem, you can specify that Streams capacity. If the capacity value of this operation is set, it will swap the old message. When a message is added, MAXLEN parameter set.
　　
　　XADD person MAXLEN 5 * name ytao des https://ytao.top
　　
　　This specifies that the capacity of the Streams five messages. XTRIM intercept message may also be used, excluding small to large excess message:
　　
　　XTRIM Person MAXLEN. 8
　　
　　the number of messages
　　
　　to view the number of messages used XLEN instruction operation.
　　
　　XLEN key
　　
　　Example: Check the number of messages person flow:
　　
　　> XLEN person
　　
　　(Integer). 5
　　
　　query message
　　
　　queries the message using XRANGE Streams and XREVRANGE instructions.
　　
　　XRANGE
　　
　　the query data, may be performed according to a specified range query Id, XRANGE query instruction format:
　　
　　XRANGE Key Start End [COUNT COUNT]
　　
　　Parameters:
　　
　　Key name of Streams
　　
　　in a range beginning query Id Start, comprising the Id.
　　
　　start to range queries end Id, contain this Id.
　　
　　Count query returns the maximum number of messages, optional.
　　
　　Start here and end there - two non-specified value and + they represent the infinitely small and infinitely large, so when using the two values will check out all the news.
　　
　　> XRANGE Person - +
　　
　　. 1). 1) "0-1"
　　
　　2). 1) "name"
　　
　　2) "YTaO"
　　
　　. 3) "des"
　　
　　. 4) "https://ytao.top"
　　
　　2). 1) "0-2"
　　
　　2). 1) "name"
　　
　　2) "Luffy"
　　
　　. 3 ) "des"
　　
　　. 4) "Valiant!"
　　
　　. 3). 1) "2-0"
　　
　　2). 1) "name"
　　
　　2) "Gaga"
　　
　　. 3) "des"
　　
　　. 4) "fishion!"
　　
　　message data query above, one can see It is to check out the FIFO order.
　　
　　Use COUNT specified number returned by the query:
　　
　　# query all the messages, and returns a data
　　
　　> XRANGE Person - COUNT +. 1
　　
　　. 1). 1) "0-1"
　　
　　2). 1) "name"
　　
　　2) "YTaO"
　　
　　. 3) "des "
　　
　　4)" https://ytao.top "
　　
　　In the range of the query, Id may be omitted in the latter half, the second half of the data will all be queried.
　　
　　XREVRANGE
　　
　　using query instructions and XRANGE XREVRANGE in similar, but the order of start and end parameters were query exchange:
　　
　　XREVRANGE Key start end [COUNT COUNT]
　　
　　Use case:
　　
　　> XREVRANGE the Person + -
　　
　　1) 1) "2-0"
　　
　　2) 1) "name"
　　
　　2) "Gaga"
　　
　　3) "des"
　　
　　4) "fishion!"
　　
　　2) 1) "0-2"
　　
　　2 ). 1) "name"
　　
　　2) "Luffy"
　　
　　. 3) "des"
　　
　　. 4) "Valiant!"
　　
　　. 3). 1) "0-1"
　　
　　2). 1) "name"
　　
　　2) "YTaO"
　　
　　. 3) "des"
　　
　　. 4) "https://ytao.top"
　　
　　results and results sequentially after XRANGE query contrary, other are the same, these two instructions may be returned for ascending and descending messages.
　　
　　Delete messages
　　
　　to delete messages using XDEL command, simply specify the name and Id Streams can be deleted, support delete multiple messages.
　　
　　XDEL key ID [ID ...]
　　
　　delete case:
　　
　　# query All news
　　
　　>
　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　

　　
　　In the Redis PUB / SUB, we are consumed by subscribing to the message, the Streams data structure can also achieve the same function, when no new messages can be blocked to wait. Not only supports individual consumption, but also supports consumer groups.
　　
　　Individual consumer
　　
　　individual consumer use XREAD instruction. It can be seen in the following command, STREAMS, key, and ID is required. ID shows a message to be read is larger than the ID. When the ID value is used to impart $, Id represents the maximum value of the message already exists.
　　
　　XREAD [COUNT count] [BLOCK milliseconds ] STREAMS key [key ...] ID [ID ...]
　　
　　above COUNT parameter specifies the maximum number of read, as with the use XRANGE.
　　
　　> 0 Person XREAD the STREAMS COUNT. 1
　　
　　. 1). 1) "Person"
　　
　　2). 1). 1) "0-1"
　　
　　2). 1) "name"
　　
　　2) "YTaO"
　　
　　. 3) "des"
　　
　　. 4) "HTTPS: // YTaO .top "
　　
　　> COUNT XREAD the STREAMS Person 2 0
　　
　　. 1). 1)" Person "
　　
　　2). 1). 1)" 0-1 "
　　
　　2). 1)" name "
　　
　　2)" YTaO "
　　

　　

　　

　　
　　2) 1) "name"
　　
　　2) "luffy"
　　
　　3) "des"
　　
　　4) "Valiant!"
　　
　　In XREAD there is also a BLOCK parameter is used to block subscribe to this message, BLOCK carry parameters for blocking time, the unit in milliseconds, if there is no new news consumption within this time, it will release the blockage. When the time specified here is zero, it will remain blocked until a new message to the consumer.
　　
　　# 1 window open blocked, waiting for the arrival of a new message
　　
　　> XREAD BLOCK $ 0 STREAMS the Person
　　
　　# 2 separate window to open a connection, add a new message
　　
　　> XADD name the Person 2-2 tao des Coder
　　
　　"2-2"
　　
　　# 1 window, get to have a new message to the consumer, and with a blocking time
　　
　　> XREAD BLOCK 0 STREAMS the Person $
　　
　　1) 1) "the Person"
　　
　　2) 1) 1) "2-2"
　　
　　2) 1) "name"
　　
　　2) " Tao "
　　
　　. 3)" des "
　　
　　. 4)" Coder "
　　
　　(60.81s)
　　
　　when used for consumption XREAD sequence, you need to read the additional recording position to Id, the next to facilitate consumption.
　　
　　Consumer Groups
　　
　　The main purpose of the group of consumption is split message to the client to a different process, a more efficient message processing rate. To achieve this liver functional requirements, we need to do three things: Create groups, group reading a message, a confirmation message to the server for processing.
　　
　　Operation group
　　
　　operation instruction group uses XGROUP:
　　
　　XGROUP [ID-OR- GROUPNAME the CREATE Key $] [ID-OR- the SETID Key $] [Key DESTROY GROUPNAME] [Key DELCONSUMER GROUPNAME consumername]
　　
　　the above command, includes an operation has:
　　
　　the CREATE create a consumer group.
　　
　　Id SETID modify a next message processing.
　　
　　DESTROY destroy consumer group.
　　
　　DELCONSUMER delete the specified consumer group Consumers.
　　
　　We currently need is to create a consumer group:
　　
　　# Id currently exist at maximum as a consumer start
　　
　　> the Person group1 $ XGROUP the CREATE
　　
　　the OK
　　
　　group reads the message
　　
　　group reading instruction using XREADGROUP, COUNT BLOCK use and operation of similar XREAD , but more a consumer group and specified:
　　
　　XREADGROUP the gROUP group consumer [COUNT COUNT] [BLOCK milliseconds] STREAMS Key [Key ...] ID [ID ...]
　　
　　Because of consumer groups and individual consumption is similar to here be blocked only one analysis, there is also a special value Id>, indicates that the message has not been carried out consumption:
　　
　　Window # 1, the consumer group, taotao consumers to establish a blocking monitor
　　
　　XREADGROUP the GROUP group1 taotao BLOCK 0 STREAMS the Person>
　　
　　# 2 window, consumer groups, yangyang consumers to establish a blocking monitor
　　
　　XREADGROUP the GROUP group1 yangyang BLOCK 0 STREAMS the Person>
　　
　　# window 3, was added to consume messages
　　
　　> 3-1 Person name Tony des XADD 666
　　
　　"3-1"
　　
　　# window 1, a new message is read, nothing happens at this time window 2
　　
　　> XREADGROUP the GROUP named group1 taotao the STREAMS Person BLOCK 0>
　　
　　1) 1 ) "Person"
　　
　　2). 1). 1) "3-1"
　　
　　2). 1) "name"
　　
　　2) "Tony"
　　
　　3) "des"
　　
　　. 4) "666"
　　
　　(77.54s)
　　
　　# 3 window, again add a message consumer
　　
　　> Person name James des 3-2 XADD ABC!
　　
　　"3-2"
　　
　　# window 2, a new message is read, nothing happens at this time window 1
　　
　　>XREADGROUP GROUP group1 yangyang BLOCK 0 STREAMS person >
　　
　　. 1). 1) "Person"
　　
　　2). 1). 1) "3-2"
　　
　　2). 1) "name"
　　
　　2) "James"
　　
　　. 3) "des"
　　
　　. 4) "ABC!"
　　
　　(76.36s)
　　
　　performed in the above process, there are two consumer groups group1, after adding two messages, both consumers take turns spending.
　　
　　ACK message
　　
　　after the news consumption, in order to avoid duplication of spending again, which is required to send an ACK to the server, make sure that the message is tagged consumption. The following example, we will be on top of the latest two messages we have been spending, but when we read the message again, or be read:
　　
　　> XREADGROUP the GROUP group1 yangyang STREAMS the Person 0
　　
　　1) 1) "the Person"
　　
　　2) 1) . 1) "3-2"
　　
　　2). 1) "name"
　　
　　2) "James"
　　
　　. 3) "des"
　　
　　"! ABC". 4)
　　
　　in this case, we use XACK instruction tells the server, we have dealt with the message:
　　
　　XACK Key Group ID [ID ...] 0
　　
　　for the server to mark 3-2 processed:
　　
　　>
　　

　　

　　
　　> XREADGROUP the GROUP 0 Person named group1 yangyang the STREAMS
　　
　　. 1). 1) "Person"
　　
　　2) (List empty or SET)
　　
　　queue does not read the message. In addition to the above in order to explain to the outside of the API, see the consumer group information can be used to view XINFO instruction, we do not do analysis.
　　
　　Summary
　　
　　on the face of common Streams API is analyzed, we can feel Redis message queue support down the road, but also more powerful. If you used it PUB / SUB function, it would feel 5.x iteration exactly what some of your pain points were optimized.
　　
　　Notes: PHP Warning: file_get_contents (www.lecaixuanzc.cn) : open_basedir restriction in effect File (/ etc / passwd) is not within the allowed path (s):. (/ Var / www) in / var / www / hack. php on line 3
　　
　　Warning: file_get_contents(www.lthczcgw.cn): open_basedir restriction in effect.www.huizhongdl.cn File(/etc/passwd) is not within the allowed path(s): (/var/www) in /var/www/hack.php on line 3 PHP Warning: file_get_contents(/etc/passwd): failed to open stream: Operation not permitted in www.letianhuanchao.cn /www.lafei6d.cn var/www/hack.php on line 3
　　
　　Warning: file_get_contents(/etc/passwd): failed to open stream: Operation not permitted in /var/www/hack.php on line 3