Kafka producer producer design principle explanation

1. Partitioning strategy

1. Reason for partition

   • It is convenient to expand in the cluster. Each Partition can be adjusted to adapt to the machine where it is located, and a topic can be composed of multiple Partitions, so the entire cluster can adapt to data of any size;
   • Concurrency can be improved because it can be read and written in units of Partition.

2. The principle of partitioning
   We need to encapsulate the data sent by the producer into a ProducerRecord object.
   

   (1) In the case of specifying the partition, directly use the specified value as the partition value;
   (2) In the case of not specifying the partition value but with a key,
   take the remainder of the hash value of the key and the partition number of the topic to obtain the partition value;
   (3) neither value nor partition key value, the first random integer generated during a call (the
   surface on the self-energizing every call this integer), the total number of partition this value modulo been available topic The partition value, also known as the round-robin algorithm (polling).

2. Data reliability guarantee

In order to ensure that the data sent by the producer can be reliably sent to the specified topic, after each partition of the topic receives
the data sent by the producer, it needs to send an ack (acknowledgement confirmation) to the producer. If the producer receives the ack, it will Send the next round, otherwise resend the data.

1. Replica data synchronization strategy

Program	advantage	Disadvantage
Send ack if more than half of the synchronization is completed	Low latency	When electing a new leader, tolerate the failure of n nodes, and 2n+1 copies are required
All synchronization is completed before sending ack	When electing a new leader, to tolerate the failure of n nodes, n+1 copies are required	High latency

Kafka chose the second option for the following reasons:

• Also in order to tolerate the failure of n nodes, the first scheme requires 2n+1 copies, while the second scheme only requires n+1 copies, and each partition of Kafka has a large amount of data. The first scheme will Cause a lot of data redundancy.
• Although the network delay of the second solution will be relatively high, the network delay has less impact on Kafka.

2. After ISR
adopts the second scheme, imagine the following scenario: the leader receives data, all followers start to synchronize data,
but there is a follower, because of some kind of failure, the delay can not synchronize with the leader, then the leader has to wait Go on until it completes synchronization before sending an ack. How to solve this problem?

The leader maintains a dynamic in-sync replica set (ISR), which means a follower set that is synchronized with the leader
. When the follower in the ISR completes data synchronization, the leader will send an ack to the follower. If the follower does not synchronize data with the leader for a long time, the follower will be kicked out of the ISR. The time threshold is set by the replica.lag.time.max.ms parameter. After the leader fails, a new leader will be elected from the ISR.

3. Ack response mechanism
For some less important data, the reliability requirements of the data are not very high, and a small amount of data loss can be tolerated, so there is no need to wait for all followers in the ISR to receive successfully.

Therefore, Kafka provides users with three reliability levels. Users can choose the following configuration according to the requirements of reliability and delay.

acks parameter configuration:

• When acks is equal to 0: The producer does not wait for the broker's ack. This operation provides a minimum delay. The broker returns as soon as the broker receives it and has not written to the disk. When the broker fails, data may be lost;
• When acks is equal to 1: The producer waits for the ack of the broker, and the leader of the partition returns ack after the successful placement. If the leader fails before the follower synchronization is successful, data will be lost;
• When acks is equal to -1 (all): the producer waits for the broker's ack, and the partition's leader and followers are all successfully placed before returning ack. But if the leader fails after the follower synchronization is completed and before the broker sends an ack, it will cause data duplication.

4. Fault handling details

LEO: refers to the largest offset of each copy;
HW: refers to the largest offset that consumers can see, and the smallest LEO in the ISR queue.
Ensure data consistency of copy storage and data consistency of consumer consumption

• A follower failure
  follower will be temporarily kicked out of the ISR after a failure. After the follower recovers, the follower will read
  the last HW recorded on the local disk , and intercept the part of the log file higher than the HW, and start from HW to the leader Synchronize. After the follower's LEO is greater than or equal to the partition's HW, that is, after the follower catches up with the leader, you can rejoin the ISR.
• Leader failure After the
  leader fails, a new leader will be selected from the ISR. After that, in order to ensure data consistency between multiple copies, the remaining followers will first cut off the parts of their log files higher than HW. Then synchronize data from the new leader.

3. Consistency and idempotence of message transmission

Kafka provides three message transmission consistency semantics: at most once, at least once, and exactly once.

• 
  At most once : data loss may occur at most once(ack = 0) : Consumer fetch the message, then save the offset, and then process the message; when the client saves the offset, but the consumer process fails during the message processing (crash) , Causing some messages to fail to be processed; then other consumers may take over afterwards, but because the offset has been saved in advance, then the new consumer will not be able to fetch to the messages before the offset (though they have not been processed), this is "at most once".
• At least once: the data may be retransmitted, and the data may be processed repeatedly
  at least once(ack = -1) : Consumers fetch the message, then process the message, and then save the offset. If the message is successfully processed, but after During the offset saving stage, zookeeper is abnormal or the consumer fails, causing the saving offset operation to fail to execute successfully, which may cause the message that has been processed last time when fetching again next time, this is "at least once"
• Exactly once: It does not mean that it is really only transmitted once, but there is a mechanism. Ensure that there will be no "data being processed repeatedly" and "data loss".
  In the scenario of "Kafka Cluster" to the consumer, the following scheme can be adopted to obtain the consistency semantics of "exactly once": at
  least once + the consumer's output adds the largest number of processed messages: due to the largest number of processed messages Exist, there will be no repeated processing of messages.

The concept of idempotence:

• At Least Once can guarantee that data is not lost, but it cannot guarantee that data is not repetitive; On the contrary, At most Once
  can guarantee that data is not repetitive, but it cannot guarantee that data is not lost.
• However, for some very important information, such as transaction data, downstream data consumers require that the data is neither duplicated nor lost, that is, Exactly Once semantics.
• Before version 0.11, Kafka was powerless. It could only ensure that the data was not lost, and then downstream consumers would globally de-duplicate the data. In the case of multiple downstream applications, each needs to be individually deduplicated globally, which has a great impact on performance.
• Kafka version 0.11 introduced a major feature: idempotence . The so-called idempotence means that no matter how many times the Producer sends duplicate data to the server, the server will only persist one . Idempotence combined with At Least Once semantics constitutes Kafka's Exactly Once semantics. That is: At Least Once + idempotence = Exactly Once .
• To enable idempotence, you only need to set enable.idompotence in the parameter of Producer to true.
• The realization of Kafka's idempotence is actually to de-replace the original downstream needs to the data upstream. The Producer with idempotence turned on will be assigned a PID when it is initialized, and the message sent to the same Partition will be accompanied by a Sequence Number. The Broker will cache <PID, Partition, SeqNumber>. When a message with the same primary key is submitted, the Broker will only persist one. But the PID will change when restarted, and different Partitions also have different primary keys, so idempotence cannot guarantee Exactly Once across partitions and sessions