Article Directory
Official document address: 2 Work Queues
In the workerallocation of tasks between the (competitive consumer model).
Prerequisites
This tutorial assumes that you have installed RabbitMQ and is running on the local host port (5672).
Work queue
In the first tutorial, we wrote a program to send and receive messages from a named queue. In this article, we will create a work queue to workerdistribute time-consuming tasks among multiple tasks.
The main idea behind the work queue (ie task queue) is to avoid executing resource-intensive tasks immediately and have to wait for it to complete (this situation is more time-consuming and affects the execution of other tasks). Instead, we schedule the task to be completed later (synchronous execution to asynchronous execution). Encapsulate the task as a message and send it to the queue. The work process running in the background will take out the message and process it. When you run multiple worker, tasks can be shared among them.
This concept is particularly useful in web applications because it is impossible to handle complex tasks in a short HTTP request window.
ready
In the previous article of this tutorial, we sent a "Hello World!"message. Now we will send a string representing a complex task. Thread.sleep()Simulate the processing time required by the task by using functions. We take the number of points in the string as the complexity, and each point represents one second of processing. For example, it Hello…takes three seconds.
We will slightly modify the Send.javacode in the previous example so that it can accept input from the command line. This program will send messages to the work queue, we name the program NewTask.java:
String message = String.join(" ", argv);
channel.basicPublish("", "hello", null, message.getBytes());
System.out.println(" [x] Sent '" + message + "'");
Our original Recv.javaprogram also needs some changes: it needs to forge one second of working time for each point in the message body. It will process the delivered messages and perform tasks, so we call it Worker.java:
DeliverCallback deliverCallback = (consumerTag, delivery) -> {
String message = new String(delivery.getBody(), "UTF-8");
System.out.println(" [x] Received '" + message + "'");
try {
doWork(message);
} finally {
System.out.println(" [x] Done");
}
};
//关闭自动消息确认机制（参见下面）
boolean autoAck = true;
channel.basicConsume(TASK_QUEUE_NAME, autoAck, deliverCallback, consumerTag -> {
});
The execution time of our simulation task:
private static void doWork(String task) throws InterruptedException {
for (char ch: task.toCharArray()) {
if (ch == '.') Thread.sleep(1000);
}
}
1Compile them as in the tutorial (using jarfiles and environment variables in the working directory CP):
javac -cp $CP NewTask.java Worker.java
On Windows:
javac -cp %CP% NewTask.java Worker.java
Round robin
One of the advantages of using task queues is the ability to easily process work in parallel. If we accumulate a backlog of work, we can add more worker, so that it is easy to scale up.
First, let's try to run two workerinstances at the same time . They will all get messages from the queue, but how do they do it? Let's see.
You need to open three consoles. Two of them will run the workerprogram. These two consoles will be our two consumers- C1and C2. For convenience, we will use environment variables for the classpath $CP(on Windows %CP%):
# shell 1
java -cp $CP Worker
# => [*] Waiting for messages. To exit press CTRL+C
# shell 2
java -cp $CP Worker
# => [*] Waiting for messages. To exit press CTRL+C
In the third console, we used to publish messages:
# shell 3
java -cp $CP NewTask First message.
# => [x] Sent 'First message.'
java -cp $CP NewTask Second message..
# => [x] Sent 'Second message..'
java -cp $CP NewTask Third message...
# => [x] Sent 'Third message...'
java -cp $CP NewTask Fourth message....
# => [x] Sent 'Fourth message....'
java -cp $CP NewTask Fifth message.....
# => [x] Sent 'Fifth message.....'
Let's see what we workersreceived:
java -cp $CP Worker
# => [*] Waiting for messages. To exit press CTRL+C
# => [x] Received 'First message.'
# => [x] Received 'Third message...'
# => [x] Received 'Fifth message.....'
java -cp $CP Worker
# => [*] Waiting for messages. To exit press CTRL+C
# => [x] Received 'Second message..'
# => [x] Received 'Fourth message....'
By default, RabbitMQ will send each message to the next consumer in turn. On average, every consumer will get the same number of messages. This way of distributing messages is called round-robin(circular). Try to get three or more workers to workeruse this method.
Message confirmation
It may take a few seconds to complete a task. What happens if one of the consumers starts a task that takes a long time, but only partially completes and then hangs. For our current code, once RabbitMQ delivers the message to the consumer, it will immediately mark it for deletion. In this case, if you kill one worker, you will lose the message it was processing. All messages sent to this specific workerbut not yet processed message will also be lost .
But we don't want to lose any tasks. If one workerdies, we hope to hand this task to the other worker.
In order to ensure that the message is never lost, RabbitMQ supports message confirmation. The consumer returns a confirmation message to tell RabbitMQ that a specific message has been received and processed, and RabbitMQ is free to delete it.
If the consumer ackdies without sending (its channel is closed, the connection is closed, or the TCP connection is lost), RabbitMQ will understand that the message has not been fully processed and will requeue it. If there are other consumers online at the same time, it will soon resend it to another consumer. In this way, even if you workerdie occasionally, you can ensure that no information is lost.
No message will time out, and RabbitMQ will redeliver the message when the consumer dies. Even if it takes a long, long time to process a message, it doesn't matter.
By default, manual confirmation is turned on. In the previous example, we autoAck=trueturned on automatic confirmation through the settings and turned off manual confirmation. Next, we turn off automatic confirmation, and use manual confirmation to confirm that the message is received successfully.
channel.basicQos(1); // 一次只接受一条未加密的消息(参见下面的内容)
DeliverCallback deliverCallback = (consumerTag, delivery) -> {
String message = new String(delivery.getBody(), "UTF-8");
System.out.println(" [x] Received '" + message + "'");
try {
doWork(message);
} finally {
System.out.println(" [x] Done");
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
};
boolean autoAck = false;
channel.basicConsume(TASK_QUEUE_NAME, autoAck, deliverCallback, consumerTag -> {
});
Using this code, we can ensure that even if you use to Ctrl+Ckill a message that is processing worker, nothing will be lost. workerSoon after this death, all unconfirmed messages will be resent.
The confirmation message must be sent on the same channel as the received message. Trying to use different channels to send confirmation messages will result in channel-level protocol exceptions. Please refer to the documentation guide on message confirmation for more information.
Forgetting the confirmation message
ForgettingbasicAckis a common mistake. This is a simple mistake, but the consequences are serious. When your client exits, the message will be resent (it may seem like a random resend), but RabbitMQ will consume more and more memory because it will not be able to release any unacknowledged messages.
In order to debug this kind of error, you can use therabbitmqctlprintmessages_unacknowledgedfield:
sudo rabbitmqctl list_queues name messages_ready messages_unacknowledged
On Windows:
rabbitmqctl.bat list_queues name messages_ready messages_unacknowledged
Information endurance
We have learned how to ensure that even if the consumer dies, the task will not be lost. However, if the RabbitMQ server stops, our tasks will still be lost.
When RabbitMQ exits or crashes, queues and messages will be lost. To ensure that the message is not lost, we need to mark both the queue and the message as persistent.
First, we need to ensure that the queue still exists after the RabbitMQ node is restarted. In order to do this, we need to declare that it is durable:
boolean durable = true;
channel.queueDeclare("hello", durable, false, false, null);
Although this command itself is correct, it does not work in our current settings. That's because we have defined a named helloqueue, which is not persistent. RabbitMQYou are not allowed to redefine an existing queue with different parameters, and an error will be returned to any program that attempts to do so. So we need to declare a queue with a different name, for example task_queue:
boolean durable = true;
channel.queueDeclare("task_queue", durable, false, false, null);
This queue declaration queueDeclarechange needs to be applied to both producer code and consumer code.
At this point, we are sure that even if RabbitMQ is restarted, the task_queuequeue will not be lost. Now, we need to mark the message as persistent-by setting the value MessageProperties(which it does BasicProperties) to PERSISTENT_TEXT_PLAIN.
import com.rabbitmq.client.MessageProperties;
channel.basicPublish("", "task_queue",
MessageProperties.PERSISTENT_TEXT_PLAIN,
message.getBytes());
Note
Marking a message as persistent does not completely guarantee that the message will not be lost. Although it tells RabbitMQ to save the message to disk, there is still a short time window when RabbitMQ receives a message and has not saved it. In addition, RabbitMQ does not execute every messagefsync(2)-it may just be saved to the cache instead of actually being written to disk. The durability guarantee is not strong, but it is sufficient for our simple task queue. If you need stronger guarantees, then you can use publisher confirmation .
Fair distribution
You may have noticed that scheduling still does not work exactly the way we want it. For example, in the two workercases, when all the odd-numbered information is heavy and the even-numbered information is light, one workerwill always be busy while the other does workeralmost nothing. Well, RabbitMQ knows nothing about this and will still distribute messages evenly.
This is because RabbitMQ only distributes messages when they enter the queue. It does not look at the number of unconfirmed messages from consumers. It just blindly sends the first nmessage to the nfirst consumer.
In order to solve this problem, we can use the basicQosmethod and set it prefetchCount = 1. This tells RabbitMQ not to workersend multiple messages to one at a time . Or, in other words, workerdon't send a new message to a message before processing and confirming it. Instead, it will send it to another one who is not too busy worker.
int prefetchCount = 1;
channel.basicQos(prefetchCount);
Pay attention to the queue size.
If everythingworkeris busy, your queue will be full. You have to pay attention to this point. You canworkersolve this problem by adding more or other strategies.
Put them together
NewTask.javaThe final code of our class:
import com.rabbitmq.client.*;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
/**
* @author wangbo
* @date 2019/10/23 11:24
*/
public class NewTask {
private final static String QUEUE_NAME = "task_queue";
public static void main(String[] args) throws IOException, TimeoutException {
//创建一个连接器连接到服务器
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
try(Connection connection = factory.newConnection()){
//创建一个通道
Channel channel = connection.createChannel();
//声明一个队列，并将队列设置为持久的
boolean durable = true;
channel.queueDeclare(QUEUE_NAME, durable, false, false, null);
//从命令行接受参数
String message = String.join(" ", args);
//发布一条消息，并将消息设置为持久的
AMQP.BasicProperties props = MessageProperties.PERSISTENT_TEXT_PLAIN;
channel.basicPublish("", QUEUE_NAME, props, message.getBytes("UTF-8"));
System.out.println(" [x] Sent '" + message + "'");
}
}
}
Worker.javaThe final code of the class:
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DeliverCallback;
import java.io.IOException;
import java.util.concurrent.TimeoutException;
/**
* @author wangbo
* @date 2019/10/22 18:25
*/
public class Worker {
private final static String QUEUE_NAME = "task_queue";
public static void main(String[] args) throws IOException, TimeoutException {
//创建一个连接器连接到服务器
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
//创建一个通道
Channel channel = connection.createChannel();
//声明一个队列，并将队列设置为持久的
boolean durable = true;
channel.queueDeclare(QUEUE_NAME, durable, false, false, null);
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
//一次只接受一条未加密的消息
int prefetchCount = 1;
channel.basicQos(prefetchCount);
//回调对象
DeliverCallback deliverCallback = (consumerTag, delivery) -> {
String message = new String(delivery.getBody(), "UTF-8");
System.out.println(" [x] Received '" + message + "'");
try {
doWork(message);
} finally {
System.out.println(" [x] Done");
//手动确认已收到一个消息
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
};
//消息自动确标志
boolean autoAck = false;
//消费者监听
channel.basicConsume(QUEUE_NAME, autoAck, deliverCallback, consumerTag -> {
});
}
private static void doWork(String task) {
for (char ch: task.toCharArray()) {
if (ch == '.') {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
}
Use message confirmation and prefetch count prefetchCountto set up the work queue. The use of persistent settings can ensure that RabbitMQ is highly available.
For more information about Channelmethods and MessageProperties, you can browse the JavaDocs online .
Now we can continue the tutorial 3and learn how to deliver the same message to many users.