Source address: http://blog.csdn.net/lizhitao/article/details/41778193
1. Kafka cluster partition replication is automatically allocated and analyzed by default
The following is an example of 4 Brokers in a Kafka cluster, creating 1 topic containing 4 Partitions and 2 Replications; the data Producer flow is shown in the figure:
(1)
(2) When 2 nodes are added to the cluster and the number of Partitions increases to 6, the distribution is as follows:
The logical rules of replica allocation are as follows:
- In a Kafka cluster, each broker has the opportunity to equally distribute the leader of the partition.
- In the above Broker Partition, the arrow points to the copy, taking Partition-0 as an example: Partition-0 in Broker1 is the Leader, and Partition-0 in Broker2 is the copy.
- Each Broker (ordered by BrokerId) in the above graphs is assigned the main Partition in turn, and the next Broker is a replica. In this way, multiple replicas follow this rule.
The copy allocation algorithm is as follows:
- Sort all N Brokers and i Partitions to be allocated.
- Assign the i-th Partition to the (i mod n)-th Broker.
- Assign the jth replica of the ith Partition to the ((i + j) mod n)th Broker.