Core concepts of Kafka cluster mode

Article directory

1. The concept of Broker|topic|partition|copy in Kafka cluster mode

1.1.Broker|topic|partition|copy concept

We have all understood the concepts of Broker, topic, and partition. Let's describe the concept of replica in detail.

The so-called copy is similar to backup. When creating a Topic topic, you need to specify the number of copies of the Topic topic. The number of copies is generally based on the number of Broker nodes in the cluster. If there are three nodes in the cluster, then there are three Broker, when creating a Topic topic, specify the number of replicas to be 3.

The number of topic copies can be understood as the backup of the partition, as shown in the figure below, the topic name is my-rc-topic, the specified number of copies is 3, and the number of partitions is 3. After the topic is successfully created, the three partitions of the topic will be One copy is created in each Broker in the cluster, and a Leader node will be elected from all replica partitions as the primary partition, and the rest of the replica partitions will be Follower nodes. For read and write operations, the Leader will periodically synchronize data to each Follower replica partition. Once the Leader replica partition hangs up, Kafka will elect from the remaining Follower replica partitions to elect a new Leader node to be the sender and consumer. read and write operations.

The copy is actually used to create multiple backup nodes in different Brokers for each partition in the Topic topic. One copy is elected as the Leader node, and the rest are Follower nodes. Even if a Broker in the cluster is broken, it will not affect Data reading.

The following figure can clearly illustrate the concept of Broker, topic, partition and copy in Kafka cluster mode.

In cluster mode, each Kafka node acts as a Broker, and Brokers are distinguished by ID numbers. Multiple partitions in a topic will create replica partitions in each Broker to ensure high data reliability.

The role of each role in the replica partition:

There are three important roles in the replica partition.

Leader: undertakes the read and write operations of the sender and the consumer, and is responsible for periodically synchronizing the message data to the follower replica partition. When the leader hangs up, a new leader is elected from multiple follower nodes through master-slave election.

Follower: Receives the data synchronized by the Leader node. When the Leader hangs up, it will vote to become the new Leader node.

Isr: Both the copy of the synchronizing data and the copy node of the data synchronization are stored in the Isr combination. When the performance of a certain node is poor, it will be kicked out of the Isr set to fully guarantee the performance of the cluster.

The concept of coherent Broker, Topic, partition, and copy:

There will be many Brokers in the cluster mode. When creating a Topic topic, you can specify how many topic partitions there are, split the message data into different partitions for storage, increase the throughput of reading and writing, and then create multiple partitions. Different replica nodes are stored in different Brokers, and data synchronization is performed between replicas to ensure high reliability of message data.

1.2. Create a Topic and specify 3 copies

1.创建一个多副本的Topic
[root@kafka-1 ~]# /data/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.20.10:2181 --replication-factor 3 --partitions 3 --topic my-rc-topic
Created topic my-rc-topic.
`--replication-factor 3`：指定副本数为3

2.查看Topic的详细信息
[root@kafka-1 ~]# /data/kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.20.10:2181 --topic my-rc-topic
Topic: my-rc-topic	PartitionCount: 3	ReplicationFactor: 3	Configs: 
	Topic: my-rc-topic	Partition: 0	Leader: 1	Replicas: 1,2,0	Isr: 1,2,0
	Topic: my-rc-topic	Partition: 1	Leader: 2	Replicas: 2,0,1	Isr: 2,0,1
	Topic: my-rc-topic	Partition: 2	Leader: 0	Replicas: 0,1,2	Isr: 0,1,2

1.3. Detailed description of Topic with multiple replicas

Topic: my-rc-topic	PartitionCount: 3	ReplicationFactor: 3	Configs: 
	Topic: my-rc-topic	Partition: 0	Leader: 1	Replicas: 1,2,0	Isr: 1,2,0
	Topic: my-rc-topic	Partition: 1	Leader: 2	Replicas: 2,0,1	Isr: 2,0,1
	Topic: my-rc-topic	Partition: 2	Leader: 0	Replicas: 0,1,2	Isr: 0,1,2
PartitionCount: 3 the number of partitions

ReplicationFactor: 3 Number of replicas

Topic: name of my-rc-topic topic

Specify how many partitions will print out as many lines, and each line is the information of a partition.

Partition: 0 The name of the partition, the first partition is 0.

Partition: 1 The name of the partition, 1 for the second partition.

Partition: 2 The name of the partition, the third partition is 2.

Partition: 0 Leader: 1 Which Broker the Leader replica node is located in the cluster, where 1 means that the Leader node of Partition 0 is on Broker-1.

Partition: 1 Leader: 2 Which Broker the Leader replica node is located in the cluster, where 2 means that the Leader node of Partition 1 is on Broker-2.

Partition: 2 Leader: 0 Which Broker the Leader replica node is located in the cluster, where 0 means that the Leader node of the second partition is on Broker-0.

Replicas: 1,2,0 Who is the Broker node where the replica is located, 1 means Broker-1, 2 means Broker-2, 0 means Broker-0.

Isr: 1,2,0 Isr set, including synchronized Broker nodes and broker nodes that have been synchronized, 1 means Broker-1, 2 means Broker-2, 0 means Broker-0, the first bit is the replica partition Leader, the second is the Controller, and the third is the Broker with no problem of synchronization.

。

After creating multiple copies, the copies of the partitions will be stored persistently on each Kafka node.

2. The concept of consuming messages from each partition in the Topic with a consumer group in the cluster mode

2.1. The concept of sub-consumer group consumption of each partition

The topic consumption concept of single partition is divided into unicast mode and multicast mode, suitable for single node mode. In cluster mode, there will be the concept of multi-copy and multi-partition. Next, let’s take a look at the concept of consumer consumption data under multi-partition .

As shown below:

The topic is divided into 4 Partition partitions, namely Partition0, Partition1, Partition2, and Partition3. The consumer has two consumer groups. There are 2 consumers in consumer group A and 4 consumers in consumer group B. Each consumer of a topic consumes different partitions, and only one consumer in the same group can consume a certain partition in a topic, which is consistent with the concept of unicast.

The concept of multi-partition consumption:

A Partition in a Topic can only be consumed by a certain consumer in a consumer group, which is consistent with the concept of unicast.
Each consumer in the same group can consume different topic partitions at the same time, but a topic can only be consumed by a certain consumer in a group.
- Consumers are one-to-many, and partitions are one-to-one.
The number of Partition partitions can determine the number of consumers in the consumption group. It is recommended that the number of consumers in the unified group should not exceed the number of partitions. It is best to keep it consistent with the number of partitions. In this way, one consumer can process one partition. Of course, there is no problem with fewer consumptions. A consumer can consume multiple partitions, but the number of consumers should not exceed the number of partitions. The extra consumers will not consume data, but will affect performance.
If a consumer in the group hangs up, the second uncle will trigger the rebalance mechanism to allow other consumers to consume the partition.

Why not let multiple consumers in the consumer group consume a partition at the same time?

The purpose of this is to ensure the sequentiality of message consumption. If multiple consumers in multiple groups consume the data of the partition at the same time, it will cause wrong consumption of messages due to network fluctuations and affect business logic.

2.2. Message sending and consumption in cluster mode

1) Message data occurs

[root@kafka-1 bin]# ./kafka-console-producer.sh --broker-list 192.168.20.10:9092,192.168.20.11:9092,192.168.20.12:9092 --topic my-rc-topic
>jiangxl nihao
>
>hahaha
>
>hello

2) Consume message data

[root@kafka-1 bin]# ./kafka-console-consumer.sh --bootstrap-server 192.168.20.10:9092,192.168.20.11:9092,192.168.20.12:9092 --from-beginning --topic my-rc-topic

3. The concept of each component in the Kafka cluster

There are three important components in Kafka cluster mode: Controller, Rabalance, HW.

3.1. Controller component

After each Broker is started, it will create a temporary serial number node in the Zookeeper cluster, and the Broker with the smallest serial number will become the Controller node in the cluster.

The role of the Controller node is as follows:

When the Leader copy in the topic partition in the cluster hangs up, the Controller node will elect a new Leader in the cluster. The election rule is to elect the leftmost Broker in the ISR set as the Leader node.
- As shown in the figure below: the leftmost in the Isr collection is the Broker number where the Leader is located. When the Leader hangs up, the Controller will delete the dead Broker from the Isr collection. At this time, there will be two Brokers left. The Broker on the side will become the new Leader, as shown in the basket.
When a Broker is added or decreased in the cluster, the Controller will synchronize the information to other Brokers.
When a partition is added or decreased in the cluster, the Controller will synchronize the information to other Brokers.

3.2.Reblance mechanism

When the consumer in the consumption group does not clearly specify the topic partition to be consumed, or the consumer suddenly hangs up when the consumer is consuming a certain partition, both of these situations will trigger the Reblance mechanism.

The Reblance mechanism will readjust which partition the consumer consumes after the relationship between the consumer and the partition changes.

Before triggering the Rebalance mechanism, there are three strategies for which partition consumers consume:

range: The list of partitions consumed by each consumer is calculated according to the formula.
- The formula before triggering: the total number of partitions / the number of consumers + 1.
- Triggered formula: total number of partitions/number of consumers.
Round Robin: All consumers consume data in the partition in turn
sticky: Sticky strategy, after triggering the Rebalance mechanism, it will make adjustments on the basis of the previous allocation, and will not change the previous allocation, only the partition associated with the failed consumer will be allocated.
- The glue policy must be enabled. If it is not started, after the Rebalance mechanism is triggered, normal partitions and consumers will be redistributed, which will degrade the performance of the cluster.

3.3. Concept of HW and LEO

HW is commonly known as high water mark, and LEO is the message position of the last message in a copy.

The HW high-water mark will be located at the position where the data in all replica partitions has been synchronized. As shown in the figure below, the HW high-water mark will be located under message data 4, and message data 5 has been written to the leader replica partition, but not yet. Synchronized to other replicas, the HW high-water mark will not move downward at this time and will not change. Only after the synchronization of all message data is completed, the HW high-water mark will be updated and moved downward. At this time, consumers can consume messages 5 , the purpose of this is to prevent message data from being lost.

Consumers can only consume up to where the HW is located. In addition, each copy has a HW, and the leader and follower are responsible for updating the status of their own HW. For the message newly written by the leader, the consumer cannot consume it immediately. The leader will wait for the message to be synchronized by all replicas in the ISR before updating the HW. At this time, the message can be consumed by the consumer. This ensures that if the broker where the leader is located fails, the message can still be obtained from the newly elected leader.