This article thoroughly understands Kafka's copy replication mechanism
Past large data memory historical memory big data
can also go to the past, large memory data read blog (click below to read the original text) https://www.iteblog.com/archives/2556.html
allow operation of the distributed system becomes simple, To a certain extent, it is an art, and this realization is usually summed up from a lot of practice. The popularity of Apache Kafka is largely due to its simplicity of design and operation. As the community adds more features, developers will go back and rethink ways to simplify complex behaviors.
A more subtle feature in Apache Kafka is its replication protocol. For workloads of different sizes on a single cluster, adjusting Kafka replication to adapt it to different situations is a bit tricky today. One of the challenges that makes this particularly difficult is how to prevent replicas from joining and exiting from the synchronized replica list (also known as ISR). From the user's perspective, this means that if a producer (producer) sends a batch of "sufficiently large" messages, then this may cause Kafka brokers to issue multiple alerts. These alerts indicate that certain topics are "under replicated", which means that the data has not been replicated to enough brokers, which increases the possibility of data loss. Therefore, it is very important for Kafka cluster to closely monitor the total number of "unreplicated" partitions. In this article, I will discuss the root cause of this behavior and how we can solve this problem.
Learn about Kafka replication mechanism in one minute
Each partition in the Kafka topic has a write-ahead log, where the messages we write to Kafka are stored. Each message here has a unique offset, which is used to identify its position in the current partition log. As shown in the figure below: 
If you want to learn about Spark, Hadoop or HBase-related articles in time, please pay attention to the WeChat official account: iteblog_hadoop
Each topic partition in Kafka has been replicated n times, where n is the replication factor of the topic. ). This allows Kafka to automatically switch to these replicas in the event of a cluster server failure, so that messages are still available in the event of a failure. Kafka replication is based on partitions, and the write-ahead logs of the partitions are replicated to n servers. Among n replicas, one replica serves as the leader, and the other replicas become followers. As the name implies, the producer can only write data to the leader partition (reading can only be performed from the leader partition), and followers only copy logs from the leader in order. 
If you want to learn about Spark, Hadoop or Hbase-related articles in time, please pay attention to the WeChat public account: iteblog_hadoop
log replication algorithm (log replication algorithm) must provide the basic guarantee if it tells the client that the message has been submitted and the current leader fails , The newly elected leader must also have this message. In the event of a failure, Kafka will select a follower from the ISR of the failed leader as the new leader of the partition; in other words, it is because the follower is keeping up with the leader's writing progress.
The leader of each partition maintains an in-sync replica (synchronized replica list, also known as ISR). When the producer sends a message to the broker, the message is first written to the corresponding leader partition, and then copied to all replicas of this partition. Only after the message has been successfully copied to all the synchronous replicas (ISR), the message is considered to be submitted. Since message replication latency is limited by the slowest synchronized copy, it is important to quickly detect the slow copy and remove it from the ISR. The details of the Kafka replication agreement will be slightly different, and this blog does not intend to discuss this topic in detail. Interested students can go here to learn more about the working principle of Kafka replication.
Under what circumstances can the replica keep up with the leader
If a replica does not keep up with the leader's log progress, it may be marked as an out-of-sync replica. I use an example to explain the meaning of catch up. Suppose we have a topic named foo, and there is only one partition, and the replication factor is 3. Suppose the copies of this partition are on brokers 1, 2 and 3, and we have submitted 3 messages on topic foo. The replica on brokers 1 is the leader, replicas 2 and 3 are followers, and all replicas are part of the ISR. Assuming replica.lag.max.messages is set to 4, this means that as long as the follower has no more than 3 messages behind the leader, it will not be deleted from the ISR. We set replica.lag.time.max.ms to 500 milliseconds, which means that as long as the follower sends a fetch request to the leader every 500 milliseconds or earlier, they will not be marked as dead and will not be removed from the ISR delete. 
If you want to learn about Spark, Hadoop or Hbase-related articles in time, please pay attention to the WeChat public account: iteblog_hadoop
Now suppose that the producer sends the next message to the leader. At the same time, a GC pause occurs on broker 3, and now the partition on each broker The situation is as follows: 
If you want to learn about Spark, Hadoop or Hbase-related articles in time, please pay attention to the WeChat public account: iteblog_hadoop
Because broker 3 is in the ISR, the latest message is not considered to be submitted until the broker 3 is removed from the ISR or the partition on broker 3 keeps up with the leader's log end offset. Note that because border 3 lags behind the leader's message is smaller than replica.lag.max.messages = 4, so it does not meet the conditions for removal from the ISR. This means that the partition on broker 3 needs to synchronize the message with offset 3 from the leader. If it does, then this replica will keep up with the leader. Assuming that broker 3 completes the GC within 100ms and keeps up with the leader's log end offset, the latest situation is as follows: 
If you want to learn about Spark, Hadoop or Hbase-related articles in time, please pay attention to the WeChat public account: iteblog_hadoop
Under what circumstances will cause a replica to lose synchronization with the leader
There are many reasons why a replica loses synchronization with the leader, mainly including:
- Slow replica: The follower replica has been unable to keep up with the leader's write progress for a period of time. One of the most common reasons for this situation is the I/O bottleneck on the follower replica, which causes it to persist logs longer than the time it takes to consume messages from the leader;
- Stuck replica: The follower replica stops getting messages from the leader for a long time. This may be because the GC is paused, or the copy is faulty;
- Bootstrapping replica: When the user increases the replication factor for a topic, the new follower replicas are not synchronized until it keeps up with the leader's log.
When the replica lags behind the leader partition, the replica is considered out of sync or lagging. In Kafka 0.8.2, the lag of the replica to the leader is measured according to replica.lag.max.messages or replica.lag.time.max.ms; the former is used to detect slow replicas, and the latter is used to detect slow replicas. Detect stuck replica (Stuck replica).How to confirm that a copy is lagging
Using replica.lag.time.max.ms to detect stuck replicas works well in all cases. It tracks the time when the follower copy did not send an acquisition request to the leader, and it can be used to infer whether the follower is normal. On the other hand, the model that uses the number of messages to detect out-of-sync slow replicas only works well when these parameters are set for a single topic or multiple topics with similar traffic patterns, but we found that it cannot be extended to production All topics in the cluster. On the basis of my previous example, the theme if foo writes at a rate of 2 msg / sec of
data, in which the leader received a single batch usually never more than three messages, then we know replica.lag this topic The .max.messages parameter can be set to 4. why? Because we write data to the leader at the maximum speed and before the follower copy replicates these messages, the follower's log lags behind the leader by no more than 3 messages. At the same time, if the follower copy of topic foo is always behind the leader by more than 3 messages, we hope that the leader deletes the slow follower copy to prevent the increase of message write delay.
This is essentially the goal of replica.lag.max.messages-to be able to detect replicas that are always out of sync with the leader. Assuming that the traffic of this topic is increasing due to the peak, the producer finally sends a batch of 4 messages to foo, which is equal to the configuration value of replica.lag.max.messages = 4. At this time, the two follower replicas will be considered out of sync with the leader, and the ISR will be removed.
If you want to learn about Spark, Hadoop or Hbase related articles in time, please pay attention to the WeChat public account: iteblog_hadoop
However, since both follower replicas are active, they will catch up with the leader's log end offset in the next fetch request and be added back to the ISR. If the producer continues to send a large number of messages to the leader, the same process described above will be repeated. This proves that unnecessary false alarms are triggered when the follower copy enters and exits the ISR.
If you want to learn about Spark, Hadoop or Hbase related articles in time, please pay attention to the WeChat public account: iteblog_hadoop
replica.lag.max.messages The core problem of the parameter is that users must guess how to configure this value, because we do not know the incoming traffic of Kafka How much will it be, especially in the case of network peaks.One parameter takes care of everything
We realize that the really important thing to detect stuck or slow replicas is the time when the replicas are out of sync with the leader. We removed the replica.lag.max.messages parameter that was set by guessing. Now, we only need to configure the replica.lag.time.max.ms parameter on the server; this parameter means the time when the replica is out of sync with the leader.
The method of detecting stuck replicas is the same as before-if the replica fails to send a fetch request within the time of replica.lag.time.max.ms, it will be treated as a dead replica and deleted from the ISR;
The mechanism for detecting slow replicas has changed-if the replica is behind the leader for more than replica.lag.time.max.ms, it is considered too slow and removed from the ISR.
Therefore, even under peak traffic, the producer sends a large number of messages to the leader, unless the replica always keeps the replica.lag.time.max.ms behind the leader, otherwise it will not randomly enter and exit the ISR.
This article is translated from Hands-free Kafka Replication: A lesson in operational simplicity: https://de.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/