Table of contents
ack response mechanism
kafka provides three response levels for users: all, leader, 0
acks ：0
This operation provides the lowest latency. When the leader of the partition receives the message, it returns ack before it is written to the disk. When the leader fails, data may be lost;
After the producer sends the message, it will not wait for any confirmation message from the broker. Although this method improves efficiency, its reliability is greatly reduced;
acks：1（leader）
After the leader of the partition is successfully downloaded, ack is returned. If the leader fails before the follower synchronization is successful, even though the leader has been successfully downloaded, the follower's synchronization progress must be lower than that of the leader. If the leader fails at this time, the follower will be lost and the leader has not yet been synchronized. That part of the data;
In this mode, there is a certain degree of reliability and efficiency, but there is still the possibility of data loss;
acks：-1（all）
ack is returned only after all the partition's leader and follower are successfully placed. However, if the leader fails after the follower synchronization is completed and before the broker sends the ack, that is, a new leader is elected, the new leader will be placed on the disk again, which will cause data duplication;
In this mode, the efficiency is the lowest, but the data reliability is the highest.
Corresponding parameters in java api
// 设置acks
properties.put(ProducerConfig.ACKS_CONFIG, "all");
// 重试次数retries，默认是int最大值，2147483647
properties.put(ProducerConfig.RETRIES_CONFIG, 3);
Two ISR collection
ISR (in- sync replica) is a set of synchronization sets maintained by Kafka for a certain partition. That is, each partition has its own ISR set. The replica in the ISR set means that the follower replica is synchronized with the leader replica. Only replicas in the ISR set are eligible to be elected as leaders. A Kafka message is considered "synchronized" only when it is received by all replicas in the ISR. This is different from the synchronization mechanism of zk. zk only needs more than half of the nodes to write before it can be considered to have been written successfully.
Imagine the following scenario
If a follower is unable to synchronize with the leader due to some failure, then if ack is selected as all, does the leader have to wait for the follower to complete synchronization before sending an ack?
Obviously not, in-sync replica set (ISR) means a set of followers that are synchronized with the leader. When the follower in the ISR completes data synchronization, the leader will send an ack to the producer. If the follower does not synchronize data with the leader for a long time, the follower will be kicked out of the ISR. The time threshold is set by the replica.lag.time.max.ms parameter. After the Leader fails, a new leader will be elected from the ISR.
The follower that was kicked out of the ISR will not be considered when electing a new leader. After the follower recovers, the follower will read the last HW recorded on the local disk and intercept the part of the log file higher than the HW, starting from the HW. Synchronize with leader. When the LEO of the follower is greater than or equal to the HW of the Partition, that is, after the follower catches up with the leader, it can rejoin the ISR.
If the same leader fails, a new leader will be selected from the ISR. After that, in order to ensure data consistency between multiple copies, the remaining followers will first cut off the parts of their log files higher than the HW, and then Synchronize data from the new leader.
LEO: refers to the maximum offset of each copy;
HW: refers to the largest offset that consumers can see and the smallest LEO in the ISR queue.
