kafka migrate or add new nodes

Migrate or add nodes

The first step is to deploy a new node

server.properties is basically the same as the old node, and the following points need to be modified:

# 这个是新节点ID,自己设置个新的就行
broker.id=2 
# 这个是新节点的ip地址
host.name=10.25.115.223

The second step is to determine the topic to be migrated

cd to kafka's bin directory to create a file named topics-to-move.json

内容如下:
{
    "topics": [
        {
            "topic": "test1"
        },
        {
            "topic": "test2"
        },
        {
            "topic": "test3"
        }
    ],
    "version": 1
}

The third step is to generate the migration configuration

Execute the following command and write the returned content to reassignment-node.json

./kafka-reassign-partitions.sh --zookeeper 10.25.115.223:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1,2" --generate

Here broker-list is the broker list modified to your own

The fourth step is to perform the migration and view the progress

./kafka-reassign-partitions.sh --zookeeper 10.25.115.223:2181 --reassignment-json-file reassignment-node.json --execute

./kafka-reassign-partitions.sh --zookeeper 10.25.115.223:2181 --reassignment-json-file reassignment-node.json --verify

Step 5 (if you are adding a new node, you can ignore this step)

Point all products to the new node. Finally, change the consumer to the new node, and you're done

question

kafka verify view status has been in is in progres

This is because the new node is not available. You can write a simple product program to test whether it can be written. The reason for the inconvenience may be because of the configuration problem of host.name. Kafka often has problems with host.name. You can try Modify advertised.host.name=10.25.115.223, (note to change to your own address), if there is no such parameter, it may be a low version.

Kafka new node does not respond nor synchronize data

The solution is the same as above

{{o.name}}

{{m.name}}