Overview
It is easy to add a server to a Kafka cluster, just assign it a unique broker ID and then start Kafka on the new server. However, no data partitions are automatically assigned to these new servers, so unless partitions are moved to them, they will not do any work until the new topic is created. Therefore, usually when adding computers to the cluster, you will need to migrate some existing data to these computers.
The data migration process is manually initiated but fully automated. Behind the scenes, Kafka will add the new server as a follower of the partition to be migrated and allow it to fully replicate the existing data in that partition. After the new server completely copies the contents of the partition and joins the synchronized copy, one of the existing copies will delete the data of its partition.
The partition reassignment tool can be used to move partitions between agents. The ideal partition allocation will ensure uniform data load and partition size among all agents. The partition redistribution tool does not have the ability to automatically study the data distribution in the Kafka cluster and move partitions around to achieve even load distribution. Therefore, the administrator must figure out which topics or partitions should be moved.
The partition reassignment tool can run in 3 mutually exclusive modes:
- generate: In this mode, given a list of topics and a list of agents, the tool generates candidate reassignments to move all partitions of the specified topic to the new agent. Given a list of topics and target agents, this option only provides a convenient way to generate a partition reassignment plan.
- execute: In this mode, the tool will start the reallocation of partitions according to the reallocation plan provided by the user. (Use the --reassignment-json-file option). This can be a custom redistribution plan hand-made by the administrator, or it can be provided using the --generate option
- verify: In this mode, the tool will verify the reallocation status of all partitions listed during the last -execute. Status can be completed successfully, failed or in progress
Partition migration:
Create a topic with three partitions and two replicas
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic topic_test
View topic
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic_test
Write migration files
vi topics-to-move.json
topics-to-move.json:
{
"topics": [{
"topic": "topic_test"}], "version":1}
Once the json file is ready, use the partition reallocation tool to generate candidate allocations
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1,2" --generate
Current partition replica assignment
{
"version":1,"partitions":[{
"topic":"topic_test","partition":2,"replicas":[2,0],"log_dirs":["any","any"]},{
"topic":"topic_test","partition":1,"replicas":[1,2],"log_dirs":["any","any"]},{
"topic":"topic_test","partition":0,"replicas":[0,1],"log_dirs":["any","any"]}]}
Proposed partition reassignment configuration
{
"version":1,"partitions":[{
"topic":"topic_test","partition":0,"replicas":[2,1],"log_dirs":["any","any"]},{
"topic":"topic_test","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{
"topic":"topic_test","partition":1,"replicas":[1,2],"log_dirs":["any","any"]}]}
The tool generates a candidate assignment and moves all partitions from topic topic_test to agent 1,2. But please note that the partition movement has not yet started at this time, it just tells you the current allocation and the proposed new allocation. The current allocation should be saved in case you want to roll it back. The new assignment should be saved in
a json file (for example, expand-cluster-reassignment.json) to be entered into the tool using the --execute option to
save the current assignment
vi old-expand-cluster-reassignment.json
old-expand-cluster-reassignment.json:
{
"version":1,"partitions":[{
"topic":"topic_test","partition":2,"replicas":[2,0],"log_dirs":["any","any"]},{
"topic":"topic_test","partition":1,"replicas":[1,2],"log_dirs":["any","any"]},{
"topic":"topic_test","partition":0,"replicas":[0,1],"log_dirs":["any","any"]}]}
vi expand-cluster-reassignment.json
expand-cluster-reassignment.json:
{
"version":1,"partitions":[{
"topic":"topic_test","partition":0,"replicas":[2,1],"log_dirs":["any","any"]},{
"topic":"topic_test","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{
"topic":"topic_test","partition":1,"replicas":[1,2],"log_dirs":["any","any"]}]}
Perform migration
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute
View the migration results,
you can see that the partition has been successfully migrated,
you can also view the results through commands
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --verify
Copy migration
First we need to restore the original partition
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file old-expand-cluster-reassignment.json --execute
Then migrate copy 2 to partition 0 and
write the migration file
vi increase-replication-factor.json
increase-replication-factor.json:
{
"version":1,"partitions":[{
"topic":"topic_test","partition":0,"replicas":[0,1,2]}]}
Perform migration
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
You can see that copy 2 has been successfully migrated to partition 1.