Kafka online expansion, and repartition

---

During the use of kakfa, due to some program bugs or some reasons, only one partition is created in the cluster, or only one partition is used to create multiple partitions.

---

1. Environment

Environment: cluster
ip: 192.168.174.44:6667 broker-id 0, 192.168.174.55:6668 broker-id 1, 192.168.174.99:6669 broker-id 2
topic: test1 single partition single copy, normal 3 partition 2 copies, official single Partition single copy

2. Testing process

1. The kafka tool creates three topics: test, normal and experiment

2. Increase the partition (difference)

Test: add three partition single copies for test1 (the number of partitions does not exceed the number of nodes)

[root@h5 bin]# ./kafka-topics.sh --zookeeper 192.168.174.55:2182  --topic normal  --describe
Topic:normal	PartitionCount:3	ReplicationFactor:2	Configs:
	Topic: normal	Partition: 0	Leader: 0	Replicas: 0,1	Isr: 0,1
	Topic: normal	Partition: 1	Leader: 1	Replicas: 1,2	Isr: 2,1
	Topic: normal	Partition: 2	Leader: 2	Replicas: 2,0	Isr: 2,0

[root@h5 bin]#./kafka-topics.sh --zookeeper 192.168.174.55:2182 --alter --topic test1 --partitions 3
Adding partitions succeeded!
[root@h5 bin]# ./kafka-topics.sh --zookeeper 192.168.174.55:2182  --topic test1  --describe           
Topic:test1	PartitionCount:3	ReplicationFactor:1	Configs:
	Topic: test1	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: test1	Partition: 1	Leader: 2	Replicas: 2	Isr: 2
	Topic: test1	Partition: 2	Leader: 0	Replicas: 0	Isr: 0

3. Offical adds partitions and allocates data.

[root@h4 bin]#  ./kafka-topics.sh --zookeeper 192.168.174.44:2181  --alter --topic official --partitions 3
Adding partitions succeeded!

[root@h4 bin]# ./kafka-reassign-partitions.sh --zookeeper 192.168.174.44:2181 --broker-list "0,1,2" --topics-to-move-json-file topic-generate.json --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"hrmwRawDatav2","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"hrmwRawDatav2","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"hrmwRawDatav2","partition":1,"replicas":[1],"log_dirs":["any"]}]}

Proposed partition reassignment configuration   ##建议的分区重新分配配置
{"version":1,"partitions":[{"topic":"hrmwRawDatav2","partition":1,"replicas":[0],"log_dirs":["any"]},{"topic":"hrmwRawDatav2","partition":0,"replicas":[2],"log_dirs":["any"]},{"topic":"hrmwRawDatav2","partition":2,"replicas":[1],"log_dirs":["any"]}]}

[root@h4 bin]# cat topic-generate.json 
{
  "topics": [
    {
      "topic": "hrmwRawDatav2"
    }
  ],
  "version": 1
}

#Increase the number of copies, replicas[this is the broker id]

[root@h4 bin]# ./kafka-reassign-partitions.sh  --zookeeper 192.168.174.44:2181  --reassignment-json-file topic-increase.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"hrmwRawDatav2","partition":0,"replicas":[0],"log_dirs":["any"]},{"topic":"hrmwRawDatav2","partition":2,"replicas":[2],"log_dirs":["any"]},{"topic":"hrmwRawDatav2","partition":1,"replicas":[1],"log_dirs":["any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

[root@h4 bin]# cat topic-increase.json 
{
	"version": 1,
	"partitions": [{
		"topic": "hrmwRawDatav2",
		"partition": 0,
		"replicas": [0,1]
	}, {
		"topic": "hrmwRawDatav2",
		"partition": 1,
		"replicas": [1,2]
	}, {
		"topic": "hrmwRawDatav2",
		"partition": 2,
		"replicas": [2,0]
	}]
}