Now the mainstream message queuing products provide a very sound mechanism to ensure the reliability of the message , the message can be done in the transfer process, even if network outages or hardware failure, but also to ensure reliable delivery of the message, the message is not lost.
The reason most of the messages are lost because the developers are not familiar with the message queue , no right to use and configure message queue caused . While the API provides a different message queue is not the same, the relevant configuration items are different, but the message to ensure reliable delivery of this child, their implementation principle is the same.
A, message loss detecting method
In general, a new system just on the line, all aspects are not very stable, the need for a run-in period, at this time, especially the need to monitor whether there has been a message to the loss of your system .
- If the IT infrastructure is relatively sound company, in general, distributed link tracking system, using similar tracking system can easily track each message.
- If no such tracking system, we can use the message queue ordering to verify that messages were lost.
Use Message orderly queue to verify the news of the loss of principle is very simple:
In Producer end, we appended to each message sent is a continuous incremental sequence number , and then to examine this end Consumer continuity of sequence numbers . If there is no message is lost, Consumer received a number of messages must be continuously increasing, or that the information received, the number of which must be on a number +1 message. If the detected number is not continuous, it is the lost message. May also be determined by the missing sequence number which message is lost, the reason facilitate further investigation.
Most client message queue support interceptor mechanism , you can use the interceptor mechanism, before sending the message number in the interceptor Producer injected into the message received message interceptor continuous numbers detected in Consumer nature, the benefits of this implementation is the message detection code does not intrude into your business code , until your system is stable, but also facilitate the detection of these logical part of closed or deleted.
Implement this detection method in a distributed system, there are several issues that need your attention:
- Like Kafka and RocketMQ such a message queue, it does not guarantee strict order on the Topic of the message can only guarantee that the partition is ordered, so we have to specify when the message of the partition , and, in each partition continuity message number of individual detectors .
- Producer if your system is multi-instance, since the order of transmission is not good coordination between a plurality of Producer, so it needs to each Producer separately generate respective message number, and requires additional identification on the Producer , in accordance with the Consumer end each Producer respectively continuity check number.
- And the same number of partitions Consumer best examples, and do Consumer partition correspondence, that would be more easily detect the continuity of the message number in the Consumer.
Second, to ensure reliable delivery of messages
Message from production to consumption process, can be divided into three stages:
1, the production stage
The most commonly used by the message queue request acknowledgment mechanism to ensure reliable delivery of the message: When you call method code message, the message queue message is sent to the client will Broker, Broker after receiving the message, the client returns an acknowledgment give In response, indicating that the message has been received. The client receives the response and sends a normal completion of a message.
Some time after the message queue in response to not receive the transmission acknowledgment will automatically retry if retries fails, return value or exception as to inform the user. When writing code to send a message, you need to pay attention to, correctly handle the return value or catch the exception , you can ensure that messages are not lost at this stage.
When synchronous transmission, as long as you can pay attention to catch the exception.
When sent asynchronously, you need to check in at the callback method . This place needs special attention, the message is lost for many reasons, we use the asynchronous send, but did not send a check result in callbacks.
2, the storage stage
Under normal circumstances, the storage phase, as long as Broker in normal operation, the problem would not lose a message appears; but if the Broker fails, such as process dies or the server is down, or the message may be lost.
If the reliability of the message is very high, you can be configured to avoid the downtime parameters Broker lost message:
- Broker for a single node, configure parameters Broker, after receiving the message, the message is written to disk and then returned to the Producer acknowledgment response , even in the event of such downtime, since the message has been written to disk, the message is not lost, after the recovery can continue to consume. For example, in RocketMQ, it is necessary to configure the brush plate SYNC_FLUSH synchronous manner flushDiskType brush disc.
- For Broker cluster is composed of a plurality of nodes, the cluster Broker needs to be configured to: at least send a message to two or more nodes, replies to give the client sends a confirmation response . So that when a Broker is down, the other can replace Broker Broker downtime, message loss does not occur.
3, message phase
Consumption stage and using similar production phase acknowledgment mechanism to ensure reliable delivery of messages, from the client Broker pull message, the user's consumer business logic execution, after successful, will be sent to the consumer Broker acknowledgment . If consumption Broker does not receive acknowledgment, the next time you pull the messages will be returned with a message, to ensure that messages are not lost in the network transmission process, not because of an error in the implementation of the client result in loss of consumer logic.
Consumption in the preparation of the code Note: Do not send consumed immediately confirmed, but should after receive a message after the completion of the implementation of all consumer business logic, and then send the consumer confirmation .