In the first issue, we wrote a program, named a queue, and sent and received messages through this queue. In this installment we will create a Work Queue to distribute tasks among multiple jobs. (I really don’t know how to translate this last sentence. Students who don’t understand it will go to the official website to check it out)
Introduction
Work Queues are also called Task Queues. The main idea is to avoid executing a resource-intensive task immediately, while waiting for it to complete. Instead, our plan is to finish the task later. We will encapsulate the task as a message and send it to the queue. A worker process running in the background will Pop (pop) the task, which will eventually execute the task. When you run a lot of jobs, tasks will be shared among them.
Note: This concept is useful in web applications for a reason: a short HTTP request window (request window) cannot handle complex requests.
Prepare
Earlier in the first issue, we sent a message containing "Hello World". In this installment, we'll send a string for complex tasks. We don't have a real exact task at hand, like "resize an image", "render a pdf", etc., so let's pretend we're busy (officially cute enough) - by using thread.sleep( ) function pretends we are busy. We'll take the number of "."s in the string as its complexity; each "." will take one second. For example, "Hello..." takes three seconds to process.
There is no official tool class here, I suggest writing a tool class to facilitate custom connections:
ConnectionUtil.java
public class ConnectionUtil {
public static Connection getConnection() throws IOException {
//定义连接工厂
ConnectionFactory factory = new ConnectionFactory();
//设置服务地址
factory.setHost("127.0.0.1");
//端口
factory.setPort(5672);
//填充账户信息
factory.setVirtualHost("{你自定的vhost}");
factory.setUsername("{你的用户名}");
factory.setPassword("{你的密码}");
//获取连接
Connection connection = factory.newConnection();
return connection;
}
}Next…
We will slightly modify the previous Sending program to send arbitrary messages from the command line. This program will be able to schedule tasks into our work queue, so let's name it NewTask.java
The official website only posted a part, I am very considerate to give you the complete! Praise me (☆▽☆)
NewTask.java
public class NewTask {
private static final String TASK_QUEUE_NAME = "task_queue";
public static void main(String[] args) throws IOException {
//获取连接
Connection connection = ConnectionUtil.getConnection();
//从连接中创建通道
Channel channel = connection.createChannel();
//创建队列
boolean durable = true;
//创建队列
channel.queueDeclare(TASK_QUEUE_NAME,durable,false,false,null);
//待传递的消息内容
String message = getMessage(args);
//传送
channel.basicPublish("","hello",null,message.getBytes());
System.out.println("[x] Sent '"+message+"'");
//关闭连接通道
channel.close();
connection.close();
}
private static String getMessage(String[] strings) {
if (strings.length<1){
return "hello world";
}
return joinStrings(strings," ");
}
private static String joinStrings(String[] strings, String delimiter) {
int length = strings.length;
if (length == 0){
return "";
}
StringBuilder words = new StringBuilder(strings[0]);
for (int i = 1;i < length; i++){
words.append(delimiter).append(strings[i]);
}
return words.toString();
}
}
(Knock on the blackboard!!!) The two methods getMessage and joinStrings help us get messages from command line parameters (args)~
Our old Receiving.java also needs some changes, it needs to receive the message and find the "." from the message, each "." will fake one second of work for us. It will process the passed message, and perform the task. So we write another class and name it Worker.java
Worker.java
public class Worker {
private static final String TASK_QUEUE_NAME = "task_queue";
public static void main(String[] args) throws IOException {
//获取连接
Connection connection = ConnectionUtil.getConnection();
//连接获取通道
Channel channel = connection.createChannel();
boolean durable = true;
//创建队列
channel.queueDeclare(TASK_QUEUE_NAME,durable,false,false,null);
final Consumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
String message = new String(body,"UTF-8");
System.out.println(" [x] Received '" + message + "'");
try {
doWork(message);
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
System.out.println("[x] Done");
}
}
};
boolean autoAck = true;
channel.basicConsume(TASK_QUEUE_NAME,autoAck,consumer);
}
private static void doWork(String task) throws InterruptedException {
for (char ch:task.toCharArray()){
if (ch=='.'){
Thread.sleep(1000);
}
}
}
}Basically, there are not too many changes. A doWork method is added. This method is a method we use to pretend that we are busy and simulate the execution time. If you don't understand the code, you will know what it does~
We'll show you the results later!
The first phase of the program did not show you the results, but it is very simple, you can play it yourself, try to see the producer first and then open the consumer, or change the order, in short, the program is here, the specific experience is different for everyone~ The first episode will not be demonstrated. The second issue needs to talk about the mystery inside, so I will show you the effect here.
results and thinking
My compiler is the mainstream IDEA, we can start two or more Workers ready to receive. I have two here.
According to our code above, the main method of NewTask needs to have input parameters to see the effect. The specific method is: run both sending and receiving to ensure that simple messages can be sent. The default is to send a hello world without any input parameters. First prove that your program is OK. Then click Edit Configurations of the NewTask program and then as shown:
See which Program argument is right! that's it! We can sequentially set
“message.”、”message..”、”message…”、”message….”、”message…..”
Remember to say that a "." will take a second
Let's see what the two workers have received.
First:
the second:
Please ignore "Done", thank you ㄟ( ▔, ▔ )ㄏ
Round-Robin Dispatching
By default, RabbitMQ will distribute each message to consumers sequentially. When the ack is received, the message will be deleted, and then the next message will be distributed to the next consumer. On average each consumer will get the same number of messages. This distribution method is called round-robin - round-robin scheduling
One of the advantages of using a Work Queue is the ability to easily parallelize work. If we accumulate a lot of work, we can add more workers so that we can easily scale up.
Doesn't it sound very tall, thinking in my heart that it must be no problem~~~ But! ! ! (Knock on the blackboard) There is a problem with this distribution model! More specific questions later
Message acknowledgment Message acknowledgment
Design a scenario: what happens when a consumer starts this long task and only partially completes it? With the current code, once RabbitMQ delivers the message to the client, it will immediately delete the message from memory. Under this premise, if we abruptly terminate the worker, the part of the task of processing this message will disappear, and what is more troubling is that all unprocessed messages sent to the worker will also disappear. Tragedy ( @ο@ )! ! !
We don't want to lose any tasks, if a worker goes down, we want to give his tasks to other workers.
In order to ensure that the message is not lost, RabbitMQ supports message acknowledgment - Message acknowledgment , an ACK will be sent back to RabbitMQ by the consumer, the message is received and processed, so that RabbitMQ can delete it freely. If a consumer dies (channel closed, connection closed, TCP connection lost...), so that it can't send ACK, RabbitMQ will know: the message is not fully processed and needs to be requeued! If there are other consumers, he will quickly transfer the message to another consumer. This way you don't have to worry about information loss.
There is no timeout, so RabbitMQ will redeliver the message when the consumer dies, even if processing the message will take a long, long time... Message acknowledgment is enabled by default, in the example we use autoACK=true (automatically send ack) way to mark it off, now we should set it to false. Once the task is complete, send an ack to the worker to confirm it.
boolean autoAck = true;
channel.basicConsume(TASK_QUEUE_NAME,autoAck,consumer);The official here also specially reminds us not to forget basicACK
Forgotten ACK (youdao translation I serve)
Missing basicAck is a very common mistake. This mistake is simple, but the consequences are serious. When your client exits (possibly some random restart), the messages will be resent, but RabbitMQ will consume more and more memory because it won't be able to free any messages that haven't been sent.
message persistence
All right! Today's main event is here! Let's talk about the message persistence that everyone is concerned about!
We have learned how to ensure that tasks are not lost if the consumer dies. However, if the RabbitMQ server stops, our tasks will still be lost. When RabbitMQ quits or crashes, we lose all queues and messages unless we tell it: "You remember them for me!" So how do we notify him?
We need to confirm two things:
1. We want to mark the queue as persistent
boolean durable = true;
//创建队列
channel.queueDeclare(”hello“,durable,false,false,null);Note here that the queue name hello has been defined, and it is not persistent. RabbitMQ does not allow redefinition of existing queues with different parameters, and will return an error to the program performing the operation. Closure can be solved, a faster way is to give another name, such as "task_queue"
boolean durable = true;
//创建队列
channel.queueDeclare(”task_queue“,durable,false,false,null);The name changed, so we should make this part of the producer and consumer code all the same.
2. We want to mark the message as persistent
At this time, we are sure that when RabbitMQ is restarted, the task_queue queue will not be lost. Now we want to mark the message as persistent - by setting the value of MessageProperties (implementing BasicProperties) to PERSISTENT_TEXT_PLAIN
channel.basicPublish("", "task_queue",MessageProperties.PERSISTENT_TEXT_PLAIN,
message.getBytes());Here is another official tip:
Note on message persistence
Marking a message as persistent does not fully guarantee that the message will not be lost. Although it tells RabbitMQ to save the message to disk, there is still a brief time gap when RabbitMQ accepts the message and doesn't save it. And RabbitMQ does not execute the fsync function for each message (specifically what this thing is, I also found out by checking Baidu). - It is only possible to save to the cache, not actually write to disk. Persistence is there, but it's not very strong, but it's enough for a simple task queue. If we need stronger guarantees, then publisher confirm can be used
Fair dispatch
One thing you may notice is that although this mode has solved a lot of problems, it still feels like something is missing. Design a scenario: There are two employees in the company, and we have many tasks, some of which are heavy and some are easy. Employee A is unlucky, and is assigned a very heavy task every time, and employee B happens to be every time. assigned to very easy tasks. At this time, our so-called fair distribution mechanism is not so fair. Because RabbitMQ sends a message every time a message enters the queue, it does not take into account the number of unacknowledged messages by consumers. This will result in employee A saving a lot of work, while employee B has little to do.
To solve this, we can use the basicQos method parameter to pass 1. This tells RabbitMQ not to give consumers more than one message at a time. In other words, don't send an employee a new message until the previous message has been processed and acknowledged. Instead, it assigns the message to a less busy consumer, such as former employee B.
int prefetchCount = 1;
channel.basicQos(prefetchCount);From the official intimate friendly reminder:
Note about queue size
If all consumers are busy, your line will be full and long. At this time, you should pay attention to this problem, such as adding more consumers, or some other strategies
Ok, so far the second issue of translation is complete. I'm not very familiar with myself, so I'm translating while I'm learning~ I hope you can correct me if there are any mistakes or incomprehensible places~