One, the background:
Our system, users can create tasks, starting tasks, but to run the task takes a long time, so the use of message queues way backstage asynchronous processing.
Is used herein RabbitMQ, the corresponding Node.js library amqplib(here is in the form of a callback: require ( "amqplib / callback_api" )).
Two, MQ processing task flow
① ② ③ ④ ⑤: HTTP request sent from the front end, the Producer (express) process, through Route -> Controller -> Function, using the amqplib sendToQueue (), uuid transmission tasks to be processed into the MQ queue. At this time, but also to modify the database, the status of the task from the "new" -> "queue".
⑥ ⑦ ⑧ ⑨ ⑩ ⑪: Consumer digestion MQ queue spit message, namely uuid task. The first task is to modify the database state is "runnning", then call the "process" module to perform complex operations, after the execution is complete, the task to modify the database state is "success" or "fail", and then return ack signal to the MQ.
Note: This requirement is relatively simple, so did not use the switch function MQ.
Third, Q & A
Q: How do MQ error handling?
MQ channel and the connection object has "error" and "close" event, need to do a good job related to logging. Especially the "error", to add reconnect mechanism to prevent errors caused because a task or MQ own reasons, affect the processing of subsequent tasks.
connection.on("error", function(err) {
// reconnect
});
channel.on("error", function(err) {
// reconnect
});Finally, according to actual needs, coupled with the global try ...... catch.
Q: How to ensure data security itself MQ message?
In order to prevent news of the loss of MQ server crashes, it is necessary to make the data persistence . Broadly divided into two:
队列持久化 + 消息持久化
channel.assertQueue(queue_name, {
durable: true
// 队列持久化
}); channel.sendToQueue(
queue_name,
Buffer.from(uuid),
{
persistent: true
// 消息持久化
},
function(err, ok) {
}
);Q: how to ensure consistency with the MQ DB data?
1, the transmission message
④ in sendToQueue (), need createConfirmChannel () using the base under such sendToQueue () third argument to have received a message MQ success callback, based on this, to combine ② the DB operation, bind for the transaction to ensure data consistency.
2, accept message
channel.consume () need to open ack mode, after the other end of all Consumer confirmation completion, then notification MQ.
channel.consume(
queue_name,
function(msg) {
const uuid = msg.content.toString();
// use uuid todo……
},
{
noAck: false
}
);Q: How to avoid MQ-prone, less fat problem
From the above how to ensure consistency with the MQ DB data? In fact, to avoid the occurrence of this problem.
But the extra to do is:
1, retry mechanism , for example, sending a failure message, a predetermined number of retries.
2, use the MQ Web console, the address of the form HTTP: // localhost: 15672 . In addition to the basic concerns of server load status, but also concerned about whether the task queue normal handling, if there stuck.
3, constructed integrally background operation and maintenance control system, the above ratio of 2 higher degree of self-defined.
4, to provide users similar to "submit a ticket" / "Feedback" / "Upload error" feature, Network access.