1. The message is lost
1. Producers failed to send
Message queues are all problems that may occur
- After the producer sends a message queue is not successfully received (network or other reason) without the knowledge of the producers, the message is lost
- After the producer sends a message queue to receive success -> Producers confirmed, but the message does not persist, the collapse of the queue, the message is lost
For this problem, three kinds of messages queue provides acknowledgment message transmission mode of the producer, for example, kafkathe acksparameter is set to greater than 0, the rabbitMQchannel set confirmmode. In rocketMQthe return message sending status code. Where rabbitMQand rocketMQalso it provides producers transaction operations .
Only some of the problems that occur before the message queue
kafkaAndrabbitMQwhen the current leader in the state of the cluster will be unavailable leader election. Inkafkathe, ifacksset to 1 (as long as the leader node receives the acknowledgment message sent by the producer that is successfully transmitted), if the current leader message received but not yet synchronized to a node from either collapsed,kafkawill had a chance to determination of non-synchronous elected leader from a node, then the message is lost, the solution is toacksset up is notallthat all the re-confirmation from node synchronization is successful. In therabbitMQmiddle, the newly added node does not mirror synchronization message before this, when the old news consumption is not fully finished, all the old node crashes, when a new node is elected as leader, will not lose all the old messages consumption (currently it seems to be no good solution).kafkaEven if all the nodes in a timely manner will not sync messages of success is not determined asynchronous node, the message is lost and the system should not be used to make a trade-off between, if you do not want the message to be lost, the entire system is not in use before the leader node recovery by parameterunclean.leader.election.enablecan be set asynchronous copy if you can become the leader node.
- Under rocketMQ cluster environment, broker master-slave mode when using asynchronous replication, if the master node crashes and the data can not be recovered, will be lost from the portion of the message has not been synchronized to the node, the solution is to use synchronous dual-write mode synchronization message, but will reduce throughput.
2. consumer spending failed
Producers and similar if the consumer fails to inform the consumer message when the message queue fails, the message is lost. kafka, rabbitMQ, rocketMQProvides similar consumer success acknowledgment mechanism to solve this problem, rabbitMQby auto_ackautomatic or manual confirmation confirming parameter settings. And rocketMQand kafkato control the consumption of information through news offset, rocketMQin the pullmode needs to maintain its own offset.
3. The queue because of health reasons from loss of data
This is well understood, for example, rabbitMQthe default message and then saved in memory, if the collapse of the queue, the message is lost naturally
2. The message sequence
1.kafka
kafkaGuarantee the same sub-region message sequence, that is to say, if you want some topicinside news all sorted, just set up a partition for the topic, which will reduce the throughput of the subject. Message key value is generally used to spread the message to the different partitions, in order to ensure the local order of the message, but such local ordering is guaranteed only when ordered to ensure the message is written to the partition, for example, producers sequential write messages 消息A, 消息B, 消息Awrite fails, 消息Bwrites success, producers retransmission 消息Aand success, this time on the order of two messages reversed, solution is set max.in.flight.requests.per.connectionto 1designated producers successfully sent upon receipt of the message not allowed to send additional information, but this approach will seriously reduce throughput before confirmation. Another problem is that kafkathe default partition uses a hashing algorithm to the message keyto reflect on the partition, if the increase in the partition map values to keynot match the previous value mapped to the partition may be. In the kafkamiddle, a partition can only be consumed by a consumer, to ensure the orderly local news consumption.
2.rocketMQ
rocketMQQueues （Message Queue）and kafkapartitions conceptually similar, so rocketMQon to ensure the orderly nature of the message queue it is also based on （Message Queue）the same kafka, like, if you want the message within a certain theme orderly, alone write queue number within this theme must be set 1, and yet kafka, as this action will significantly reduce the throughput of the subject. rocketMQAfferent custom message when sending MessageQueueSelectorthe local message ordering guarantees the production, in the consumer message pushby mode MessageListenerOrderlyconsumption guarantee the order.
3.rabbitMQ
For rabbitMQ, I did not find the relevant information, personal guess, rabbitMQordering the same message queue should be guaranteed, but most will increase in news judgment ordering producers and consumers
3. Repeat message
Almost all of the messages in the queue have failed to provide a guarantee of non-repetition of the message, and the message and the message is repeated almost lost a second election issue, in most cases we have chosen to ensure that messages are not lost but tolerate part of the message is repeated, the most widely used solution message is repeated manner verifies the message to ensure the consumer or the consumer side idempotency