Redis study notes 02- message queue and queue delay

EDITORIAL: Redis message queue is not a professional message queue, there is no guarantee ACK, not a particularly large number of advanced features, if there are high demands on the reliability of the message, they give it up now.

1.Redis message queue

Redis is achieved by the asynchronous message queue list data structure inside. Through rpush, lpushand lpop, rpopconstituting the different effects.
For example, rpushand lpopthe operation is similar to the combined effect constituting a queue (FIFO); a rpushand rpopcan be configured effects stack (LIFO), but generally in the form of message queues are queues using a combination of effects.

>rpush hobby basketball football tennis
(integer) 3
>lpop hobby
"basketball"

>lpop hobby
"football"

>lpop tennis
"tennis"

>lpop hobby
(nil)

Client gets the message queue by pop operation, and then processed, processing over again then get the message, and then deal with, has been circulating, this is the queue consumer client life cycle.

As mentioned above, if the queue is empty, pop operation can not get the data, the client has been caught in an endless loop of pop produced a bunch of useless empty polling, while consuming the resources Redis client and service side, how do?

Firstly, you can think of that might make the request queue is empty, the sleepcommand thread to stop. Better approach is through blpopand brpopbefore the substitute lpopand rpopthat a blocking read (there is no data in the queue, and immediately go to sleep; when the data arrives, and immediately re-request) way of dealing with the delay because the sleep command brings.

But this will have a new problem arises: When the list is a long time without data, the client thread will always be blocked, the formation of idle connections, but in this case, in order to save server resources are usually disconnects this link. Client blpop/ brpoperror is reported.

So you want to write client-side catch exceptions when consumers try again.

2. Latency Queuing

Delay occurs in the queue in order to solve the distributed development, client lock issue failed while processing the request. In this way the current conflict is thrown into the request queue of another process to avoid delay lock conflict.

Redis delay in the queue is zsetimplemented (ordered list), the basic mode is used zset key score value. Often in the message sequence into a delay queue as a string zsetof value, as the expiration time of the message score, followed by a plurality of threads (guaranteed availability) to poll zsetto obtain due to process tasks. Taking into account the concurrent grab the task, the task is likely to be executed multiple times.

def delay(msg):
    msg.id = str(uuid4()) #序列化保证value的唯一性
    value = json.dumps(msg)
    retry_ts = time.time() + 5 # 5s后重试 
    redis.zadd('delay-queue',retry_ts,value)

def loop():
    while True:
        # 最多取一条
        value_list = redis.zrangebyscore("delay-queue",0,time.time(),start=0,num=1): 
        if not value_list:
            time.sleep(1)
            continue
        value = value_list[0]
        # 将取到的消息从redis消息队列中删除
        success = redis.zrem("delay-queue",value)
        # 多进程并发下，只有一个线程可以抢到消息
        if success:
            msg = json.loads(value)
            handle_msg(msg)

Redis The zremmethod is to ensure that multi-threaded competition for mission-critical confusion does not occur.
Note that in the execution handle_msg () when an exception must be captured to avoid problems lead to a single task cycle quit unexpectedly.