First, the persistent queue
Statement queue queue_declare method prototype:
channel.queue_declare(queue='', passive=False, durable=False,
exclusive=False, auto_delete=False,
arguments=None):
queue: queue name
durable: Whether to persist, the default queue statement is False, that is stored in memory, if rabbitmq restart will be lost.
If you want to exist after a reboot is necessary to make persistent queue, to save Erlang comes Mnesia database after restart rabbitmq reads the database.
exclusive: whether exclusive, the default is False, not exclusive. There are two effects:
A: connection.close () whether the queue will be automatically deleted when the connection is closed;
Two: whether the queue is private private, if not exclusive, you can use two consumers have access to the same queue, no problem;
If it is exclusive and will lock the current queue, allowing only current consumers can access, other channels channel is not accessible, mandatory access will be reported if an exception: ShutdownSignalException: channel error
Generally equal to true, then only one queue for a consumer to consume the scene.
auto_delete: whether to automatically delete, if the queue to be deleted automatically defaults to False when the last consumer is disconnected.
When can RabbitMQ Management, the number of consumers viewing a queue, when consumers = 0, that is not the task of consumers, the queue will be automatically deleted
arguments: messages in the queue when it will automatically be deleted? Message TTL (x-message-ttl ): Set the lifetime of all messages in the queue ( unified set for the entire life cycle of all message queue ),
Can also be specified separately when the release message is a message the remaining survival time, in milliseconds, similar to redis in ttl, survival time is up, the message will be removed from the team, note that the message is deleted, rather than queue is delete, characteristics Features = TTL, set the expiration time for a single message: properties = pika.BasicProperties (.........)
Auto Expire (x-expires): When a specified time in the queue is not accessed (consume, basicGet, queueDeclare ...) will be deleted, Exp Features =
Max the Length (X-max-length): defines the maximum message queue length, than the first specified length will be removed a few, similar to a fixed set of mongodb, e.g. save the latest message 100, the Feature Lim =
max the length bytes (X-max-length-bytes): defining a maximum queue amount of space, generally limited by the size of memory, disk, Lim Features B =
Dead Letter Exchange (X-Dead-Letter-Exchange): delete the message when the queue length is greater than the maximum length, expired or the like, from the queue message pushed to a specified switch to rather than discarded, Features = DLX
Dead Letter routing key (X-Dead-Letter-routing-key): the deleted message pushed to specified switch queue specified routing keys go, = DLK the Feature
the maximum priority (the X--max-priority): priority queue, the queue declaration to define the highest priority value (maximum value defined generally not too much), specify the message at the time of dissemination of information Priority, a higher priority (higher values) of the first message is consumed,
Lazy MODE (X = the lazy-Queue-MODE): Lazy Queues: first message stored on disk, not in memory, when the consumer began to consume when it is loaded into memory
Master locator (x-queue-master -locator)
Statement on queue, if you use the same set of parameters is declared, you can not use other parameters to declare either delete the queue again deleted, you can use the command line deletion may also be deleted on the RabbitMQ Management, or to re-queue a name .
Queue persistence:
Restart RabbitMQ server (by rabbitmqctl stop_app shut down the server, rabbitmqctl start_app restart the server), you can log RabbitMQ Management-> Queues in. If the queue is set to persist, you can see the queue before the declaration there.
Second, news endurance of
Message acknowledgment mechanism:
If an exception occurs when the consumer message queue nor the message, the server can not know whether this message is successful, then the loss will happen?
No, because there is a message confirmation mechanism: in fact, the entry message is taken out of consumers, there is no message in the queue (but will be temporarily stored elsewhere). But rabbitMQ confirmation message-consumer success or failure, if the message is an exception, that fails, a message from another place this entry back into the queue; if successful will delete this message how long depending on the configuration.
News endurance of:
If the message server is down, the server queue and whether the message will be saved?
If you do not enable message persistence (the default), the message is stored in memory, and message queues downtime will be lost.
If the queue, the message persistence, it is saved in erlang own database, restart the server after recovery queues and messages.
Setting message persistence, persistence must set the queue, the queue is not persistent or else, the message persistence, queues do not exist, there is are there any message. Message persistence is necessary to switch persistent, persistent queue, the message persistence, persistence in order to ultimately achieve the purpose.
Set as a single message persistence: publish message, the parameter setting properties = pika.BasicProperties (delivery_mode = 2), 2 for the persistence of a non-persistent.
channel.basic_publish(exchange='',
routing_key="task_queue",
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # make message persistent
))
Message TTL message remaining lifetime
of all messages for the same set of queues unified statement cycle: a unified set expiration time of all messages in the queue, for example, set for 10 seconds, the message queue is cleared after 10 seconds
arguments.put("x-message-ttl", 10000);
// declare the message queue specified expiration time queue
channel.queue_declare(QUEUE_NAME, false, false, false, arguments);
Auto Expire automatically expire
x-expires for how long when no consumer can access the queue when the queue is automatically deleted, you can set a delay time, as a producer only start after 10 seconds of the queue will be deleted, or start a producer, and then start a consumer, the consumer runs 10 seconds after the end of the queue will be deleted
max length the maximum length
x-max-length: Specifies the length of the queue for, if not specified, can be considered infinitely long , such as specifying the queue length is 4, when more than four message previous messages are deleted, the following bits to message Proton
max length bytes fragment
x-max-length-bytes: specifies queue storage space for messages size, when the data before the maximum is reached will be removed to make room
maximum priority biggest priority
x-max-priority: set the priority of the message the greater the priority value, the more the consumer in advance.
NA priority under normal circumstances
the Hello RabbitMQ:. 1
the Hello RabbitMQ: 2
the Hello RabbitMQ:. 3
the Hello RabbitMQ:. 4
the Hello RabbitMQ:. 5
using the order of priority opposite
the Hello RabbitMQ:. 5
the Hello RabbitMQ:. 4
the Hello RabbitMQ:. 3
the Hello RabbitMQ: 2
the Hello RabbitMQ: 1
Third, examples
One of the producers, consumers of Message Queue Mode:
Producer:
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
message = ' '.join(sys.argv[1:]) or "Hello World!"
channel.basic_publish(exchange='',
routing_key='task_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # This message persistence
))
print(" [x] Sent %r" % message)
connection.close()
consumer:
#!/usr/bin/env python
import pika
import time
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
time.sleep(body.count(b'.'))
print(" [x] Done")
# Message Confirmation: After the completion of consumer spending, send a confirmation message to the server
ch.basic_ack(delivery_tag=method.delivery_tag)
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.basic_qos(prefetch_count=1)
channel.basic_consume(callback,
queue='task_queue',
# No_ack = False # The default is False
)
channel.start_consuming()
1. If consumers take to the message from the queue, but the consumer has failed? How to ensure that this provision will be refunded to the message queue, the message can be retrieved by the other?
Consumers in the consumer success, sending a message to confirm; in the code, which must be achieved in two places:
- In consumer callback function, sending an acknowledgment message: ch.basic_ack (delivery_tag = method.delivery_tag)
- no_ack consumption parameters are the channel.basic_consume uses the default value False
2. If rabbitMQ server hung up, how to ensure that the server is restarted with the queue of messages are not lost?
Persistent queue, and the message persistence. Both must be persistent. Code:
- When you declare a queue, use the Queue parameter durable = True persistence:
- channel.queue_declare(queue='task_queue', durable=True)
- Setting queue Message TTL message remaining lifetime
- When a message producer or released, the parameter properties = pika.BasicProperties (delivery_mode = 2) that the message persistence: channel.basic_publish (exchange = '',
routing_key='task_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # This message persistence
))
Four, fairness dispatch
You may have noticed that the work schedule is still not fully in accordance with our requirements. For example, there are two workers in the case, when all the strange information are very heavy, even when very little information, a staff member would have been very busy, another staff hardly do any work. So, RabbitMQ not know anything about this, and will even send a message.
This happens because RabbitMQ scheduling message only when the message into the queue. It does not consider the number of consumers unconfirmed message. It just blindly distribute messages to the n-th bit n consumers.
To solve this problem, we can use basic.qos methods and set prefetch_count = 1 . This tells RabbitMQ time not to send multiple messages to staff. Or in other words, do not send a new message to staff, until it is processed and confirmed the previous message. Instead, it will dispatch it to the next is not still busy staff.
channel.basic_qos(prefetch_count=1)
Example:
Producer: new_task.py
#!/usr/bin/env python
import sys
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
message = ' '.join(sys.argv[1:]) or 'Hello World'
channel.basic_publish(exchange='',
routing_key='task_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # ensure that the message is persistent
))
print(" [x] Sent %r" % message)
connection.close()
Consumers: worker.py
#!/usr/bin/env python
import time
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
time.sleep(body.count(b'.'))
print(" [x] Done")
ch.basic_ack(delivery_tag=method.delivery_tag)
channel.basic_consume(callback,
queue='hello')
channel.basic_qos(prefetch_count=1)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
Use message acknowledgment and prefetch_count , you can set up a work queue. Even RabbitMQ Restart, persistence options also let the task continue to exist.
First, the persistent queue
Statement queue queue_declare method prototype:
channel.queue_declare(queue='', passive=False, durable=False,
exclusive=False, auto_delete=False,
arguments=None):
queue: queue name
durable: Whether to persist, the default queue statement is False, that is stored in memory, if rabbitmq restart will be lost.
If you want to exist after a reboot is necessary to make persistent queue, to save Erlang comes Mnesia database after restart rabbitmq reads the database.
exclusive: whether exclusive, the default is False, not exclusive. There are two effects:
A: connection.close () whether the queue will be automatically deleted when the connection is closed;
Two: whether the queue is private private, if not exclusive, you can use two consumers have access to the same queue, no problem;
If it is exclusive and will lock the current queue, allowing only current consumers can access, other channels channel is not accessible, mandatory access will be reported if an exception: ShutdownSignalException: channel error
Generally equal to true, then only one queue for a consumer to consume the scene.
auto_delete: whether to automatically delete, if the queue to be deleted automatically defaults to False when the last consumer is disconnected.
When can RabbitMQ Management, the number of consumers viewing a queue, when consumers = 0, that is not the task of consumers, the queue will be automatically deleted
arguments: messages in the queue when it will automatically be deleted? Message TTL (x-message-ttl ): Set the lifetime of all messages in the queue ( unified set for the entire life cycle of all message queue ),
Can also be specified separately when the release message is a message the remaining survival time, in milliseconds, similar to redis in ttl, survival time is up, the message will be removed from the team, note that the message is deleted, rather than queue is delete, characteristics Features = TTL, set the expiration time for a single message: properties = pika.BasicProperties (.........)
Auto Expire (x-expires): When a specified time in the queue is not accessed (consume, basicGet, queueDeclare ...) will be deleted, Exp Features =
Max the Length (X-max-length): defines the maximum message queue length, than the first specified length will be removed a few, similar to a fixed set of mongodb, e.g. save the latest message 100, the Feature Lim =
max the length bytes (X-max-length-bytes): defining a maximum queue amount of space, generally limited by the size of memory, disk, Lim Features B =
Dead Letter Exchange (X-Dead-Letter-Exchange): delete the message when the queue length is greater than the maximum length, expired or the like, from the queue message pushed to a specified switch to rather than discarded, Features = DLX
Dead Letter routing key (X-Dead-Letter-routing-key): the deleted message pushed to specified switch queue specified routing keys go, = DLK the Feature
the maximum priority (the X--max-priority): priority queue, the queue declaration to define the highest priority value (maximum value defined generally not too much), specify the message at the time of dissemination of information Priority, a higher priority (higher values) of the first message is consumed,
Lazy MODE (X = the lazy-Queue-MODE): Lazy Queues: first message stored on disk, not in memory, when the consumer began to consume when it is loaded into memory
Master locator (x-queue-master -locator)
Statement on queue, if you use the same set of parameters is declared, you can not use other parameters to declare either delete the queue again deleted, you can use the command line deletion may also be deleted on the RabbitMQ Management, or to re-queue a name .
Queue persistence:
Restart RabbitMQ server (by rabbitmqctl stop_app shut down the server, rabbitmqctl start_app restart the server), you can log RabbitMQ Management-> Queues in. If the queue is set to persist, you can see the queue before the declaration there.
Second, news endurance of
Message acknowledgment mechanism:
If an exception occurs when the consumer message queue nor the message, the server can not know whether this message is successful, then the loss will happen?
No, because there is a message confirmation mechanism: in fact, the entry message is taken out of consumers, there is no message in the queue (but will be temporarily stored elsewhere). But rabbitMQ confirmation message-consumer success or failure, if the message is an exception, that fails, a message from another place this entry back into the queue; if successful will delete this message how long depending on the configuration.
News endurance of:
If the message server is down, the server queue and whether the message will be saved?
If you do not enable message persistence (the default), the message is stored in memory, and message queues downtime will be lost.
If the queue, the message persistence, it is saved in erlang own database, restart the server after recovery queues and messages.
Setting message persistence, persistence must set the queue, the queue is not persistent or else, the message persistence, queues do not exist, there is are there any message. Message persistence is necessary to switch persistent, persistent queue, the message persistence, persistence in order to ultimately achieve the purpose.
Set as a single message persistence: publish message, the parameter setting properties = pika.BasicProperties (delivery_mode = 2), 2 for the persistence of a non-persistent.
channel.basic_publish(exchange='',
routing_key="task_queue",
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # make message persistent
))
Message TTL message remaining lifetime
of all messages for the same set of queues unified statement cycle: a unified set expiration time of all messages in the queue, for example, set for 10 seconds, the message queue is cleared after 10 seconds
arguments.put("x-message-ttl", 10000);
// declare the message queue specified expiration time queue
channel.queue_declare(QUEUE_NAME, false, false, false, arguments);
Auto Expire automatically expire
x-expires for how long when no consumer can access the queue when the queue is automatically deleted, you can set a delay time, as a producer only start after 10 seconds of the queue will be deleted, or start a producer, and then start a consumer, the consumer runs 10 seconds after the end of the queue will be deleted
max length the maximum length
x-max-length: Specifies the length of the queue for, if not specified, can be considered infinitely long , such as specifying the queue length is 4, when more than four message previous messages are deleted, the following bits to message Proton
max length bytes fragment
x-max-length-bytes: specifies queue storage space for messages size, when the data before the maximum is reached will be removed to make room
maximum priority biggest priority
x-max-priority: set the priority of the message the greater the priority value, the more the consumer in advance.
NA priority under normal circumstances
the Hello RabbitMQ:. 1
the Hello RabbitMQ: 2
the Hello RabbitMQ:. 3
the Hello RabbitMQ:. 4
the Hello RabbitMQ:. 5
using the order of priority opposite
the Hello RabbitMQ:. 5
the Hello RabbitMQ:. 4
the Hello RabbitMQ:. 3
the Hello RabbitMQ: 2
the Hello RabbitMQ: 1
Third, examples
One of the producers, consumers of Message Queue Mode:
Producer:
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
message = ' '.join(sys.argv[1:]) or "Hello World!"
channel.basic_publish(exchange='',
routing_key='task_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # This message persistence
))
print(" [x] Sent %r" % message)
connection.close()
consumer:
#!/usr/bin/env python
import pika
import time
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
time.sleep(body.count(b'.'))
print(" [x] Done")
# Message Confirmation: After the completion of consumer spending, send a confirmation message to the server
ch.basic_ack(delivery_tag=method.delivery_tag)
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.basic_qos(prefetch_count=1)
channel.basic_consume(callback,
queue='task_queue',
# No_ack = False # The default is False
)
channel.start_consuming()
1. If consumers take to the message from the queue, but the consumer has failed? How to ensure that this provision will be refunded to the message queue, the message can be retrieved by the other?
Consumers in the consumer success, sending a message to confirm; in the code, which must be achieved in two places:
- In consumer callback function, sending an acknowledgment message: ch.basic_ack (delivery_tag = method.delivery_tag)
- no_ack consumption parameters are the channel.basic_consume uses the default value False
2. If rabbitMQ server hung up, how to ensure that the server is restarted with the queue of messages are not lost?
Persistent queue, and the message persistence. Both must be persistent. Code:
- When you declare a queue, use the Queue parameter durable = True persistence:
- channel.queue_declare(queue='task_queue', durable=True)
- Setting queue Message TTL message remaining lifetime
- When a message producer or released, the parameter properties = pika.BasicProperties (delivery_mode = 2) that the message persistence: channel.basic_publish (exchange = '',
routing_key='task_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # This message persistence
))
Four, fairness dispatch
You may have noticed that the work schedule is still not fully in accordance with our requirements. For example, there are two workers in the case, when all the strange information are very heavy, even when very little information, a staff member would have been very busy, another staff hardly do any work. So, RabbitMQ not know anything about this, and will even send a message.
This happens because RabbitMQ scheduling message only when the message into the queue. It does not consider the number of consumers unconfirmed message. It just blindly distribute messages to the n-th bit n consumers.
To solve this problem, we can use basic.qos methods and set prefetch_count = 1 . This tells RabbitMQ time not to send multiple messages to staff. Or in other words, do not send a new message to staff, until it is processed and confirmed the previous message. Instead, it will dispatch it to the next is not still busy staff.
channel.basic_qos(prefetch_count=1)
Example:
Producer: new_task.py
#!/usr/bin/env python
import sys
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
message = ' '.join(sys.argv[1:]) or 'Hello World'
channel.basic_publish(exchange='',
routing_key='task_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode = 2, # ensure that the message is persistent
))
print(" [x] Sent %r" % message)
connection.close()
Consumers: worker.py
#!/usr/bin/env python
import time
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
time.sleep(body.count(b'.'))
print(" [x] Done")
ch.basic_ack(delivery_tag=method.delivery_tag)
channel.basic_consume(callback,
queue='hello')
channel.basic_qos(prefetch_count=1)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
Use message acknowledgment and prefetch_count , you can set up a work queue. Even RabbitMQ Restart, persistence options also let the task continue to exist.