Outline
The main idea of the work queue (job queue) is to avoid performing resource-intensive tasks immediately, but have to wait for it to complete, instead of our schedule tasks to complete this task in the future, we just want to send messages to the queue, running in the background the final task of the working process pop perform this task.
Loop Scheduling
The advantage of using the task queue that can easily parallelized work (consumption this task)
news_task.py
1 #!/usr/bin/env python 2 # -*- coding:utf-8 -*- 3 4 import pika 5 import sys 6 7 credentials = pika.PlainCredentials('admin', 'admin123456') 8 connection = pika.BlockingConnection(pika.ConnectionParameters(host='192.168.1.6', credentials=credentials)) 9 channel = connection.channel() 10 channel.queue_declare(queue='hello') 11 message = ' '.join(sys.argv[1:]) or "Hello World!" 12 channel.basic_publish(exchange='', 13 routing_key='hello', 14 body=message) 15 16 print('[x]发送%s' % message) 17 connection.close()
worker.py
1 #!/usr/bin/env python 2 # -*- coding:utf-8 -*- 3 4 import pika 5 import time 6 7 credentials = pika.PlainCredentials('admin', 'admin123456') 8 connection = pika.BlockingConnection(pika.ConnectionParameters(host='192.168.1.6', credentials=credentials)) 9 channel = connection.channel() 10 channel.queue_declare('hello') 11 12 13 def callback(ch, method, properties, body): 14 print('[x] Received %r' % body) 15 time.sleep(20) 16 print("[x] Done") 17 18 19 channel.basic_consume(queue='hello', 20 auto_ack=True, 21 on_message_callback=callback) 22 print(' [*] Waiting for messages.') 23 channel.start_consuming()
In this case three open console, a news_task, two worker, operating results
At this point we can see at a single worker first mission was complete when, if followed by a new task, to pick up a second worker1. So here RabbitMQ will in turn send each message to the next consumer, if the average number of messages received each consumer is the same, this is the way to distribute news cycle.
Message acknowledgment
When performing a task may take some time, if one worker only completed part of the task and be killed, in this case, RabbitMQ once to send a message out, it will immediately delete the mark, which would be lost under processing of the message. But do not want to lose any messages, RabbitMQ support message acknowledgment, the consumer sends back an acknowledgment (acknowledgement) inform RabbitMQ has been received, dealt with this task, then you can delete RabbitMQ.
If the worker process died suddenly, RabbitMQ know the news is not completely processed and re-queue. If there are other consumers it will be distributed to other consumers, ensure that the task is not lost
By default, it is in an open message acknowledgment, but at the top we set up auto_ack = True is closed to confirm.
Equitable distribution
Problems caused by:
If you do not consider RabbitMQ load, then just turn distribute evenly, then the message, such as a particularly busy work process, and the other a worker process is almost nothing to do, this will consume a lot of time.
To overcome this problem: we can configure basic_qos method prefetch_count = 1, so if RabbitMQ is to tell me that the process did not work processed tasks do not give me a task, to other work processes hair.
The following two graphs can illustrate the task unfinished tasks will be issued to the worker process recv2
Message persistence fair distribution code +
sender.py
1 #!/usr/bin/env python 2 # -*- coding:utf-8 -*- 3 4 import time 5 import sys 6 import pika 7 8 credentials = pika.PlainCredentials('admin', 'admin123456') 9 connection = pika.BlockingConnection(pika.ConnectionParameters(host='192.168.1.6', credentials=credentials)) 10 channel = connection.channel() 11 #If the RabbitMQ server suddenly stops, the task will be lost, durable = True on all configured queues lasting 12 # queue_declare here affirmed queue when producers and consumers must be set 13 channel.queue_declare (Queue = ' task_queue ' , durable = True) 14 Message = ' ' .join (the sys.argv [. 1:]) or " Hello World% S " % the time.time () 15 channel.basic_publish (= Exchange '' , 16 routing_key = ' task_queue ' , . 17 = body Message, 18 is #The message is marked as a persistent 2 = delivery_mode . 19 Properties = pika.BasicProperties (delivery_mode = 2,) # message persistence 20 is ) 21 is 22 is Print ( ' [X]% R & lt Sent ' % Message) 23 is Connection.close ()
recv1.py
1 #!/usr/bin/env python 2 # -*- coding:utf-8 -*- 3 4 import time 5 import pika 6 7 credentials = pika.PlainCredentials('admin', 'admin123456') 8 connection = pika.BlockingConnection(pika.ConnectionParameters(host='192.168.1.6', credentials=credentials)) 9 channel = connection.channel() 10 channel.queue_declare('task_queue', durable=True) . 11 12 is 13 is DEF the callback (CH, Method, Properties, body): 14 Print ( " Received% S " % body) 15 the time.sleep (20 is ) 16 Print ( ' [X] the Done ' ) . 17 # confirmation must be establishing a transmission on the same channel (channel protocol would result if the same channel is not abnormal) 18 is ch.basic_ack (delivery_tag = method.delivery_tag) . 19 20 is 21 is channel.basic_qos (= prefetch_count. 1 ) 22 is channel.basic_consume (Queue = ' task_queue ' , 23 is on_message_callback=callback, 24 ) 25 26 print(' [*] Waiting for messages. To exit press CTRL+C') 27 channel.start_consuming()
recv2.py
1 #!/usr/bin/env python 2 # -*- coding:utf-8 -*- 3 4 import pika 5 6 credentials = pika.PlainCredentials('admin', 'admin123456') 7 connection = pika.BlockingConnection(pika.ConnectionParameters(host='192.168.1.6', credentials=credentials)) 8 channel = connection.channel() 9 channel.queue_declare('task_queue', durable=True) 10 11 12 is DEF the callback (CH, Method, Properties, body): 13 is Print ( " Received% S " % body) 14 Print ( ' [X] the Done ' ) 15 # confirmation must be established transmission (if not the same on the same channel will result in abnormal channel protocol channel) 16 ch.basic_ack (delivery_tag = method.delivery_tag) . 17 18 is . 19 channel.basic_qos (= prefetch_count. 1 ) 20 is channel.basic_consume (Queue = ' task_queue ' , 21 is on_message_callback = the callback, 22 is ) 23 24 print(' [*] Waiting for messages. To exit press CTRL+C') 25 channel.start_consuming()
Restart rabbitmq task remains
