场景
线上很多 kafka 的 topic 的副本数为1,这样的设置丧失了 kafka 高可用的特性,所以我们需要把 topic 的副本数至少设置为2(当前3台 kafka)
我们使用 kafkat 这个工具,这个工具是 ruby 写的,所以我们先安装 ruby 环境
参考 https://ruby-china.org/wiki/rvm-guide
这时候你应该已经由了 gem 命令
继续参考https://github.com/airbnb/kafkat 把 kafkat 装上, 并配置好你的 kafka 环境参数到 ~/.kafkatcfg
实操
我们修改 topic 的 replicas 为2
[root
@
001
kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
Topic Partition Leader Replicas ISRs
IM_APP_BINSPECT_MSG_TOPIC
0
2
[
2
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
1
3
[
3
] [
3
]
IM_APP_BINSPECT_MSG_TOPIC
2
1
[
1
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
3
2
[
2
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
4
3
[
3
] [
3
]
IM_APP_BINSPECT_MSG_TOPIC
5
1
[
1
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
6
2
[
2
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
7
3
[
3
] [
3
]
IM_APP_BINSPECT_MSG_TOPIC
8
1
[
1
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
9
2
[
2
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
10
3
[
3
] [
3
]
IM_APP_BINSPECT_MSG_TOPIC
11
1
[
1
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
12
2
[
2
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
13
3
[
3
] [
3
]
IM_APP_BINSPECT_MSG_TOPIC
14
1
[
1
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
15
2
[
2
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
16
3
[
3
] [
3
]
IM_APP_BINSPECT_MSG_TOPIC
17
1
[
1
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
18
2
[
2
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
19
3
[
3
] [
3
]
[root
@
001
kafka]# kafkat set-replication-factor IM_APP_BINSPECT_MSG_TOPIC --newrf
2
--brokers
1
,
2
,
3
This operation executes the following assignments:
Topic Partition Replicas
IM_APP_BINSPECT_MSG_TOPIC
0
[
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
1
[
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
2
[
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
3
[
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
4
[
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
5
[
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
6
[
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
7
[
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
8
[
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
9
[
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
10
[
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
11
[
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
12
[
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
13
[
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
14
[
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
15
[
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
16
[
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
17
[
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
18
[
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
19
[
3
,
2
]
Proceed (y/n)?
y
Beginning.
Started.
[root
@
001
kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
Topic Partition Leader Replicas ISRs
IM_APP_BINSPECT_MSG_TOPIC
0
2
[
2
,
1
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
1
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
2
1
[
1
,
3
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
3
2
[
2
,
1
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
4
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
5
1
[
1
,
3
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
6
2
[
2
,
1
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
7
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
8
1
[
1
,
3
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
9
2
[
2
,
1
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
10
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
11
1
[
1
,
3
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
12
2
[
2
,
1
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
13
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
14
1
[
1
,
3
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
15
2
[
2
,
1
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
16
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
17
1
[
1
,
3
] [
1
]
IM_APP_BINSPECT_MSG_TOPIC
18
2
[
2
,
1
] [
2
]
IM_APP_BINSPECT_MSG_TOPIC
19
3
[
3
,
2
] [
3
]
|
解释
上面一共执行了3个命令
- kafkat partitions IM_APP_BINSPECT_MSG_TOPIC 查看 topic 的分区情况
- kafkat set-replication-factor IM_APP_BINSPECT_MSG_TOPIC --newrf 2 --brokers 1,2,3 重新设置副本数为2 --newrf 2 , 你有几个 broker就在--brokers 后面指定
- kafkat partitions IM_APP_BINSPECT_MSG_TOPIC 检查是否生效
根据 topic 数据量最后完成的时间也不同(大概几分钟)
这是最后的完成情况
[root
@
001
kafka]# kafkat partitions IM_APP_BINSPECT_MSG_TOPIC
Topic Partition Leader Replicas ISRs
IM_APP_BINSPECT_MSG_TOPIC
0
2
[
2
,
1
] [
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
1
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
2
1
[
1
,
3
] [
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
3
2
[
2
,
1
] [
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
4
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
5
1
[
1
,
3
] [
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
6
2
[
2
,
1
] [
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
7
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
8
1
[
1
,
3
] [
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
9
2
[
2
,
1
] [
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
10
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
11
1
[
1
,
3
] [
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
12
2
[
2
,
1
] [
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
13
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
14
1
[
1
,
3
] [
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
15
2
[
2
,
1
] [
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
16
3
[
3
,
2
] [
3
,
2
]
IM_APP_BINSPECT_MSG_TOPIC
17
1
[
1
,
3
] [
1
,
3
]
IM_APP_BINSPECT_MSG_TOPIC
18
2
[
2
,
1
] [
2
,
1
]
IM_APP_BINSPECT_MSG_TOPIC
19
3
[
3
,
2
] [
3
,
2
]
|