In the previous article, the program was implemented to send and receive messages from a named queue. This article will create a work queue to assign some time-consuming tasks to multiple workers.
The main idea of the work queue is to avoid processing a resource-consuming task immediately and waiting for it to complete. Instead, we can add it to the plan list and perform these tasks later. We divide the task into a message and send it to the queue. The background work program will execute the task immediately after receiving the message. When running multiple actuators, tasks will be shared among them.
This concept is more practical in web applications, for complex tasks that cannot be completed in a short http request.
1. Prepare
The above is to send a message containing "Hello World". Now let's send a string representing a complex task. We do not have a real task here, such as the task of modifying the size of the image and rendering the pdf file. Here we simulate a scene where the task is busy (using the Thread.sleep () function). Here we use the number of dots in the string class to represent the complexity of the task, and each dot takes up one second of processing time. For example, a forged task described with "Hello ..." will take three seconds.
Let ’s modify the Send.java code above to send arbitrary messages from the client. This program will assign tasks to our work list, named NewTask.java:
The message sending part is as follows:
String message = getMessage(argv);
channel.basicPublish("", "hello", null, message.getBytes());
System.out.println(" [x] Sent '" + message + "'");
Get the message content from the operating parameters:
private static String getMessage(String[] strings){
if (strings.length < 1)
return "Hello World!";
return joinStrings(strings, " ");
}
private static String joinStrings(String[] strings, String delimiter) {
int length = strings.length;
if (length == 0) return "";
StringBuilder words = new StringBuilder(strings[0]);
for (int i = 1; i < length; i++) {
words.append(delimiter).append(strings[i]);
}
return words.toString();
}
The old receiver also needs to be slightly modified: a comma in the message body represents a one-second task, and the receiver will receive the message and then execute the task. Here renamed to Work.java:
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Received '" + message + "'");
doWork(message);
System.out.println(" [x] Done");
}
Then simulate the time spent executing tasks:
private static void doWork(String task) throws InterruptedException {
for (char ch: task.toCharArray()) {
if (ch == '.') Thread.sleep(1000);
}
}
2. Polling scheduling
A big advantage of the task queue is that it can easily arrange work. If some work is being backlogged in the background queue, it can be solved by adding more workers.
First, let's run two worker instances (C1 and C2) at the same time. They will get messages from the queue at the same time. For details, see below:
shell1$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
[*] Waiting for messages. To exit press CTRL+C
shell2$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
[*] Waiting for messages. To exit press CTRL+C
Then publish the task (run the sender):
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask First message.
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask Second message..
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask Third message...
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask Fourth message....
shell3$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
NewTask Fifth message.....
Then check what tasks our workers performed:
shell1$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
[*] Waiting for messages. To exit press CTRL+C
[x] Received 'First message.'
[x] Received 'Third message...'
[x] Received 'Fifth message.....'
shell2$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar
Worker
[*] Waiting for messages. To exit press CTRL+C
[x] Received 'Second message..'
[x] Received 'Fourth message....'
By default, RabbitMQ will send each message to each consumer as a poll, and evenly send the message to each consumer. This method of allocation management is called polling, and can also test the situation of multiple workers.
3. Message response mechanism
It takes a few seconds to complete a task. You must be curious, what if a consumer takes a long time to start a task and crashes when it reaches a certain part. In our current code, after pushing a message to consumers, RabbitMQ will immediately delete the message. In this case, if we kill a worker, then we will lose the message that the worker is processing the task (the task is not processed and completed), and we will also lose all the messages that have been sent to this consumer and have not been processed. .
However, we do not want to lose this part of the message, we hope that such messages can be sent to other workers again.
In order to ensure that messages will never be lost, RabbitMQ supports a message reply mechanism. When the consumer receives the message and completes the task, he sends a confirmation command to the RabbitMQ server, and then RabbitMQ deletes the message.
If a consumer hangs up while sending confirmation messages, RabbitMQ will treat the service as not completed, and then send the service that executed the message to another consumer. In this way, even if a worker hangs, the message will not be lost.
It is not judged by timeout here. Only when a consumer is disconnected, RabbitMQ will resend the message that the consumer did not return confirmation to other consumers. It takes a long time to process a certain task instantly, and there is no problem here.
The message response mechanism is turned on by default. In the above example, we explicitly turned it off (autoAck = true), then the program should now be modified as follows:
QueueingConsumer consumer = new QueueingConsumer(channel);
boolean autoAck = false;
channel.basicConsume("hello", autoAck, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
//...
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
This will ensure that even if you kill the worker, there will be no loss of information. After the worker is killed, all unconfirmed messages will be resent.
Error-prone points:
Many people will forget to call the basicAck method. Although this is a very simple error, it is often fatal. After the consumer exits, the message will be resent, but because some unconfirmed messages cannot be released, RabbitMQ will consume more and more memory.
To be able to debug this kind of error, you can use rabbitmqctl to print out the messages_unacknowledged field.
$ sudo rabbitmqctl list_queues name messages_ready messages_unacknowledged
Listing queues ...
hello 0 0
...done.
4. Informative endurance
We have learned the fault tolerance mechanism when the consumer hangs or the task is killed. Let's see how to ensure that the message is not lost when the RabbitMQ service is stopped.
When RabbitMQ exits or is down, queues and messages will be lost. Of course, there are two places to pay attention to to solve this type of problem: persistent storage of queues and messages.
First, we want to ensure that RabbitMQ will never lose the message queue, then we need to declare it as persistent storage:
boolean durable = true;
channel.queueDeclare("hello", durable, false, false, null);
Although the operation here is correct, it will not take effect here because the queue named "hello" has been created before (non-persistent), and now it already exists. RabbitMQ does not allow you to redefine an existing message queue. If you try to modify some of its attributes, your program will report an error. So, here you need to change a message queue name:
boolean durable = true;
channel.queueDeclare("task_queue", durable, false, false, null);
Both producers and consumers need to use the queueDeclare method to specify persistent properties.
Now we can ensure that even if RabbitMQ restarts, the task queue will not be lost. Below I will implement message persistence (by setting the property MessageProperties. PERSISTENT_TEXT_PLAIN, where MessageProperties implements the BasicProperties interface).
import com.rabbitmq.client.MessageProperties;
channel.basicPublish("", "task_queue",
MessageProperties.PERSISTENT_TEXT_PLAIN,
message.getBytes());
Marking message persistence does not guarantee 100% that the message will not be lost. Although RabbitMQ will write the message to the disk, but receiving the message from RabbitMQ to write to the disk, RabbitMQ restarts during this short period of time Messages written to disk will still be lost. In fact, this is the case. After RabbitMQ receives the message, it will first write the message to the memory buffer, not directly write a single message to the disk in real time. The persistence of messages is not robust, but it is sufficient for simple task queues. If you need a very robust persistence solution, then you can use publisher confirms (more on how to use it later).
5. Fair task distribution strategy
You may notice that sometimes RabbitMQ cannot distribute messages as you expected. For example, if there are two workers, the tasks corresponding to the odd-numbered messages are very time-consuming, and the tasks corresponding to the even-numbered messages can be executed quickly. In this case, one of the workers will always be busy, and the other worker will hardly do the task. RabbitMQ will not do anything to deal with this phenomenon and still push messages evenly.
This is because RabbitMQ is pushed to the consumer after the message is pushed by the producer, and it does not check the number of messages that have not received confirmation from the consumer. It will only distribute all N messages to N consumers.
To solve this problem, we can use basicQos to set how many messages the consumer will receive at the same time. Set to 1 here, indicating that RabbitMQ does not send more than one message to consumers at the same time. In this way, we can ensure that after processing a task and sending confirmation information, RabbitMQ will push new messages to it. If there are new messages between them, it will be pushed to other consumers. All consumers are processing tasks, then they will wait.
int prefetchCount = 1;
channel.basicQos(prefetchCount);
Note the size of the message queue:
If all workers are in a busy state, your message queue may be too long (memory or disk bottlenecks). You need to pay as much attention to this information as possible, and you can add workers as appropriate.
6. The final implementation of the code
Sending end:
import java.io.IOException;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.MessageProperties;
public class NewTask {
private static final String TASK_QUEUE_NAME = "task_queue";
public static void main(String[] argv)
throws java.io.IOException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
//指定队列持久化
channel.queueDeclare(TASK_QUEUE_NAME, true, false, false, null);
String message = getMessage(argv);
//指定消息持久化
channel.basicPublish( "", TASK_QUEUE_NAME,
MessageProperties.PERSISTENT_TEXT_PLAIN,
message.getBytes());
System.out.println(" [x] Sent '" + message + "'");
channel.close();
connection.close();
}
//...
}
Receiving end:
import java.io.IOException;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.QueueingConsumer;
public class Worker {
private static final String TASK_QUEUE_NAME = "task_queue";
public static void main(String[] argv)
throws java.io.IOException,
java.lang.InterruptedException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("localhost");
Connection connection = factory.newConnection();
Channel channel = connection.createChannel();
//指定队列持久化
channel.queueDeclare(TASK_QUEUE_NAME, true, false, false, null);
System.out.println(" [*] Waiting for messages. To exit press CTRL+C");
//指定该消费者同时只接收一条消息
channel.basicQos(1);
QueueingConsumer consumer = new QueueingConsumer(channel);
//打开消息应答机制
channel.basicConsume(TASK_QUEUE_NAME, false, consumer);
while (true) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println(" [x] Received '" + message + "'");
doWork(message);
System.out.println(" [x] Done" );
//返回接收到消息的确认信息
channel.basicAck(delivery.getEnvelope().getDeliveryTag(), false);
}
}
//...
}
Using the message reply mechanism and prefetchCount can implement a work queue. The persistence option allows tasks to be queued and messages will not be lost even after RabbitMQ restarts.
For more applications on Channel and MessageProperties, you can refer to the official Java API documentation:
http://www.rabbitmq.com/releases/rabbitmq-java-client/current-javadoc/
Final summary:
1. The consumer opens the message response mechanism on the channel and ensures that the confirmation information of the received message can be returned, so as to ensure that the consumer will not lose the message if it fails.
2. The server and the client must specify the persistence of the queue and the persistence of the message, so that RabbitMQ can be restarted, and the queue and the message will not be.
3. Specify the number of messages received by consumers to avoid the problem of unreasonable utilization of resources that occurs when messages are evenly pushed.
