python3 asynchronous message queue RQ processing

rqIt is Redis Queuethe abbreviation, based on a Redissimple, lightweight tool asynchronous message queue.

If the user initiates a request that takes a long time in the website, the server will return a timeout if it is synchronized. At this time, you need to use asynchronous requests. After the user initiates the request, the server throws the job to another process for execution, and then immediately returns it to the user. The user then obtains the execution progress and execution result of the job through polling or other methods.

rqThe workereffect is equivalent to starting a new daemon, listening to new tasks come in it will be executed, and user-initiated request immediately returned.

worker

Task monitoring, daemon process.

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-

from redis import Redis
from rq import Worker, Connection


QUEUES = ['default']
redis_url = 'redis://127.0.0.1:6379/15'
redis_connection = Redis.from_url(redis_url)


def run_worker():
    with Connection(redis_connection):
        worker = Worker(QUEUES)
        worker.work()


if __name__ == '__main__':
    run_worker()

send

Send message queue.

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-
	
from rq import Queue
from task import task
from redis import Redis


redis_url = 'redis://127.0.0.1:6379/15'
redis_connection = Redis.from_url(redis_url)
queue = Queue(connection=redis_connection)


if __name__ == '__main__':
    job = queue.enqueue(task, name='异步队列')
    print(job.get_status())

task

实际执行的任务。

#!/usr/bin/env python3
# -*- coding: UTF-8 -*-

import time


def task(name):
    time.sleep(2)
    print(name)
    time.sleep(2)
    return name

Start worker.

(demo) MacBook:RQ zhangyi$ python worker.py 
12:30:57 Worker rq:worker:3f2e384e636941939f7a054f20ad3ba5: started, version 1.6.1
12:30:57 Subscribing to channel rq:pubsub:3f2e384e636941939f7a054f20ad3ba5
12:30:57 *** Listening on default...
12:31:00 default: task.task(name='异步队列') (b539426a-9776-4b11-810d-a2d73c779c18)
异步队列
12:31:04 default: Job OK (b539426a-9776-4b11-810d-a2d73c779c18)
12:31:04 Result is kept for 500 seconds

Perform sending tasks.

(demo) MacBook:RQ zhangyi$ python send.py 
queued
(demo) MacBook:RQ zhangyi$

Check the execution of the job.

If the function is executed normally, it returns the return of the job, if there is an exception, it returns None, if the job is not executed, it also returns None.

print(job.result)

Obtaining the status of the job queued is still in the queue, failed: execution failed, finished completed.

print(job.get_status())

Set an id for the job. If there is no set_id operation, the job id will be a random unique string.

job.set_id('my_id')

Get the id of the job.

my_id = job.get_id()

Convert the job instance into a dictionary.

print(job.to_dict())

Delete the job from redis.

print(job.delete())

Cancel the job, even though the job has been executed, it can also be cancelled.

print(job.cancel())

Returns whether there is a job with that id.

from rq import job
job.Job.exists(my_id, redis_conn)

Create an instance of the job id.

from rq import job
my_job = job.Job(my_id, redis_conn)

If we are using the project Flask, a special assembly Flask-RQ2available.

python3 asynchronous message queue RQ processing

Guess you like