Question 1: How the message is not lost
How to protect the messaging middleware 100% message delivery success? In the configuration may be achieved by kafka ack parameters:
- Parameter 0: producer does not wait for ack, if not yet written beoker returns, cause data loss.
- Parameter 1: return ack leader after successfully written, without waiting for a successful synchronization follower;
- -1 parameters: After all the waiting disks folleower colonies returned successfully.
ack mechanism does not guarantee 100% the message is not lost, for the following reasons:
If we send a message to every producer, every MQ persisted to disk, and then initiate a callback ack or a nack. In this case we are not very high throughput MQ, because every time the message is persisted to disk. This action is written to disk is very slow. In this high concurrency scenarios are not acceptable, the throughput is too low.
So MQ persistence to achieve real disk, is handled by an asynchronous call, he is a certain mechanism, such as: wait until there are thousands of messages, when a one-time brush disk to it. Rather than to each message, a brush disk.
So comfirm asynchronous mechanism is actually a listening mechanism, in order to ensure the system's high throughput, thus leading to still can not 100% guarantee the message is not lost, because even with a mechanism to confirm the message in memory has not brush MQ disk-to-disk on down, and I could not handle.
Solution: news ahead of a scheduled task persistence +
Flow Description:
- Prior to re-order service producers to deliver the messages, first message is persisted to Redis or DB, recommendations Redis , high performance. Status message for transmission.
- confirm whether the mechanism listen for messages sent successfully? As ack success message, delete Redis in this message.
- If nack message if unsuccessful, this may be selected according to a service message retransmission. You can also delete the message, determined by their business.
- Here added a timed task to pull a certain time interval, or to send the message status in this state to indicate that the service is not received orders ack success message.
- Regular tasks will make delivery of the message compensatory. This time if MQ callbacks ack successfully received, then Redis in this message to delete.
Question 2: how to ensure there is no duplication send a message?
In order to achieve the Producer idempotency, Kafka introduced Producer ID (i.e. PID) and Sequence Number.
- PID. Each new Producer in the initialization time will be assigned a unique PID, the PID is not visible to the user.
- Sequence Numbler. (For each PID, the transmission data for each Producer <Topic, Partition> Number corresponds to a monotonically increasing from 0 Sequence
Kafka may be multiple producers, results in a message at the same time, but Kafka who only need to ensure that within each producer idempotent message on it, all of the introduction of the PID to identify different producers.
For Kafka, it is to be solved by the producer to send a message of power and other issues. That is necessary to distinguish between whether to repeat each message.
Kafka by adding a message for each Sequence Numbler, each message distinguished by Sequence Numbler. Each message corresponds to a partition, the partition of different messages generated impossible to repeat. All Sequence Numbler corresponding to each partition
Broker end stored in the cache of this seq number, for each message received, if its sequence number one more than the number Broker cache accept it, otherwise it is discarded. This can be achieved repeat the message presented. However, to ensure that only a single Producer Exactly Once the semantics for the same <Topic, Partition> a. Producer does not guarantee a topic with a different partion idempotent.
