注意
- 此性能测试在虚拟机上测试！
- kafka基准测试，消息发送，消息消费
- 集群规模为三个broker
- 此测试对topic不同分区进行测试
- 2.1版本kafka已经取消了多少线程发送的测试，所以不对发送线程进行归类测试
集群环境
机器：测试机器163，164，165
kafka Info：3节点broker，jvm 1g，日志168小时(测试时，单独对topic进行了日志存储时间的设置，为的是自动清除数据，测试机器硬盘小)
kafka version：2.1.1
虚拟机配置：8 Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz 8个逻辑核 64位
1 physical id : 0
1 address sizes : 43 bits physical, 48 bits virtual
1 physical id : 2
1 address sizes : 43 bits physical, 48 bits virtual
1 physical id : 4
1 address sizes : 43 bits physical, 48 bits virtual
1 physical id : 6
1 address sizes : 43 bits physical, 48 bits virtual
1 physical id : 8
1 address sizes : 43 bits physical, 48 bits virtual
1 physical id : 10
1 address sizes : 43 bits physical, 48 bits virtual
1 physical id : 12
1 address sizes : 43 bits physical, 48 bits virtual
1 physical id : 14
1 address sizes : 43 bits physical, 48 bits virtual
虚拟机核不做过多赘述
Linux XXGL-T-TJSYZ-app-026 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
测试环境
机器：172.24.29.163（虚拟机）
虚拟机配置：8 Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
性能测试工具
kafka自带测试脚本：kafka-producer-perf-test.sh、kafka-consumer-perf-test.sh
生产者测试
分区数不同
分区数分别为3、9、15，消息数都为：100W，消息大小：100byte，副本数都为1
创建Topic：
注意：topic名称之前不能有空格
./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test_producer_3_1 -partitions 3 -replication-factor 1
./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test_producer_6_1 -partitions 6 -replication-factor 1
./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test_producer_9_1 -partitions 9 -replication-factor 1
./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test_producer_12_1 -partitions 12 -replication-factor 1
./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test_producer_15_1 -partitions 15 -replication-factor 1
设置topic消息保存时间：
修改消息保存时间为1分钟
./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test_producer_3_1 --add-config retention.ms=60000 --alter
./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test_producer_9_1 --add-config retention.ms=60000 --alter
./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test_producer_15_1 --add-config retention.ms=60000 --alter
生产消息：
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_3_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 811688.311688 records/sec (77.41 MB/sec), 5.51 ms avg latency, 228.00 ms max latency, 5 ms 50th, 12 ms 95th, 15 ms 99th, 29 ms 99.9th.
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_6_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 827129.859388 records/sec (78.88 MB/sec), 5.05 ms avg latency, 180.00 ms max latency, 3 ms 50th, 13 ms 95th, 17 ms 99th, 23 ms 99.9th.
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_9_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 840336.134454 records/sec (80.14 MB/sec), 6.88 ms avg latency, 169.00 ms max latency, 4 ms 50th, 16 ms 95th, 32 ms 99th, 35 ms 99.9th.
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_12 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 769230.769231 records/sec (73.36 MB/sec), 6.08 ms avg latency, 184.00 ms max latency, 4 ms 50th, 18 ms 95th, 43 ms 99th, 66 ms 99.9th.
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_15_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 847457.627119 records/sec (80.82 MB/sec), 5.38 ms avg latency, 170.00 ms max latency, 4 ms 50th, 12 ms 95th, 21 ms 99th, 32 ms 99.9th.
结果：
| 节点数 | 分区数 | 副本数 | 每秒发送消息数 records/sec |
发送速率 MB/sec | 平均延迟 | 最大延迟 | 50% | 95% | 99% |
|---|---|---|---|---|---|---|---|---|---|
| 3 | 3 | 1 | 811688 | 77.41 | 5.51 ms | 228.00 ms | 5ms | 12ms | 15ms |
| 3 | 6 | 1 | 827130 | 78.88 | 5.05 ms | 180.00 ms | 3ms | 13ms | 17ms |
| 3 | 9 | 1 | 840336 | 80.14 | 6.88 ms | 169.00 ms | 4ms | 16ms | 32ms |
| 3 | 12 | 1 | 769231 | 73.36 | 6.08 ms | 184.00 ms | 4ms | 18ms | 43ms |
| 3 | 15 | 1 | 847458 | 80.82 | 5.38 ms | 170.00 ms | 4ms | 12ms | 21ms |
副本数不同
分区数都为9，消息数都为：100W，消息大小：100byte，副本数分别为1、2、3
创建Topic：
./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test_producer_9_2 -partitions 9 -replication-factor 2
./kafka-topics.sh -create -zookeeper 172.24.29.163:2181/kafka211 -topic test-producer-9-3 -partitions 9 -replication-factor 3
设置topic消息保存时间：
./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test_producer_9_2 --add-config retention.ms=60000 --alter
./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test_producer_9_3 --add-config retention.ms=60000 --alter
生产消息：
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_9_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 803858.520900 records/sec (76.66 MB/sec), 3.00 ms avg latency, 194.00 ms max latency, 3 ms 50th, 6 ms 95th, 10 ms 99th, 12 ms 99.9th.
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_9_2 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 803212.851406 records/sec (76.60 MB/sec), 7.69 ms avg latency, 183.00 ms max latency, 6 ms 50th, 19 ms 95th, 23 ms 99th, 24 ms 99.9th.
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_9_3 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 795544.948290 records/sec (75.87 MB/sec), 13.01 ms avg latency, 194.00 ms max latency, 11 ms 50th, 31 ms 95th, 45 ms 99th, 56 ms 99.9th.
结果：
| 节点数 | 分区数 | 副本数 | 每秒发送消息数 records/sec |
发送速率 MB/sec | 平均延迟 ms | 最大延迟 ms | 50% ms | 95% ms | 99% ms | 99.9% ms |
|---|---|---|---|---|---|---|---|---|---|---|
| 3 | 9 | 1 | 803859 | 76.66 | 3.00 | 194 | 3 | 6 | 10 | 12 |
| 3 | 9 | 2 | 803213 | 76.60 | 7.69 | 183 | 6 | 19 | 23 | 24 |
| 3 | 9 | 3 | 803213 | 75.87 | 13.01 | 194 | 11 | 31 | 45 | 56 |
消息大小不同
分区数都为9，消息数都为：100W，副本数分别为1，消息大小分别为：10byte、100byte、1000byte
生产消息：
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_9_1 --throughput 100000000 --record-size 10 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 869565.217391 records/sec (8.29 MB/sec), 3.12 ms avg latency, 180.00 ms max latency, 3 ms 50th, 6 ms 95th, 16 ms 99th, 22 ms 99.9th.
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_9_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 812347.684809 records/sec (77.47 MB/sec), 4.67 ms avg latency, 169.00 ms max latency, 3 ms 50th, 12 ms 95th, 13 ms 99th, 21 ms 99.9th.
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_9_1 --throughput 100000000 --record-size 1000 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
1000000 records sent, 233972.859148 records/sec (223.13 MB/sec), 124.23 ms avg latency, 282.00 ms max latency, 150 ms 50th, 238 ms 95th, 260 ms 99th, 280 ms 99.9th.
结果：
| 节点数 | 分区数 | 副本数 | 消息大小 byte | 每秒发送消息数 records/sec |
发送速率 MB/sec | 平均延迟 ms | 最大延迟 ms | 50% ms | 95% ms | 99% ms | 99.9% ms |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 3 | 9 | 1 | 10 | 869565 | 8.29 | 3.12 | 180 | 3 | 6 | 16 | 22 |
| 3 | 9 | 1 | 100 | 812348 | 77.47 | 4.67 | 169 | 3 | 12 | 13 | 21 |
| 3 | 9 | 1 | 1000 | 233973 | 223.13 | 124.23 | 282 | 150 | 283 | 260 | 280 |
消费者测试
线程数不同
为方便测试，修改消息保存时间为1h：
./kafka-configs.sh --zookeeper 172.24.29.163:2181/kafka211 --entity-type topics --entity-name test_producer_3_1 --add-config retention.ms=600000 --alter
生产消息：
节点数：3、分区数：3、副本数：1、消息大小：1000byte
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_3_1 --throughput 100000000 --record-size 1000 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
消费消息：
注意使用不同的消费组消费同一topic
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-1 --timeout 60000 --threads 1
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 16:56:13:720, 2020-10-21 16:56:15:128, 363.5883, 258.2304, 1000000, 710227.2727, 14, 1394, 260.8238, 717360.1148
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-2 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 16:56:33:331, 2020-10-21 16:56:34:686, 363.9135, 268.5709, 1000341, 738259.0406, 11, 1344, 270.7690, 744301.3393
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-3 --timeout 60000 --threads 20
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 16:56:48:238, 2020-10-21 16:56:49:648, 363.5883, 257.8641, 1000000, 709219.8582, 13, 1397, 260.2637, 715819.6135
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-4 --timeout 60000 --threads 30
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 16:57:06:195, 2020-10-21 16:57:07:610, 363.5883, 256.9529, 1000000, 706713.7809, 12, 1403, 259.1506, 712758.3749
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-5 --timeout 60000 --threads 40
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 16:58:05:760, 2020-10-21 16:58:07:201, 363.9708, 252.5821, 1000401, 694240.8050, 15, 1426, 255.2390, 701543.4783
解释：
start.time：测试开始时间,
end.time：测试结束时间,
data.consumed.in.MB：消费的所有消息的大小(MB),
MB.sec：每秒传输的数据量(MB),
data.consumed.in.nMsg：测试总共消费的消息数,
nMsg.sec：一秒内消费的消息数,
rebalance.time.ms：均衡耗时(ms),
fetch.time.ms：拉取耗时(ms),
fetch.MB.sec：每秒拉取数据(MB),
fetch.nMsg.sec：每秒拉取消息数
结果：
| 线程数 | 消息总大小(MB) data.consumed.in.MB |
每秒传输数据大小(MB) MB.sec |
消费消息总数 data.consumed.in.nMsg |
每秒消费消息数 nMsg.sec |
均衡耗时(ms) rebalance.time.ms |
拉取耗时(ms) fetch.time.ms |
每秒拉取数据(MB) fetch.MB.sec |
每秒拉取消息数 fetch.nMsg.sec |
|---|---|---|---|---|---|---|---|---|
| 1 | 363.5883 | 258.2304 | 1000000 | 710227.2727 | 14 | 1394 | 260.8238 | 717360.1148 |
| 10 | 363.5883 | 268.5709 | 1000000 | 738259.0406 | 11 | 1344 | 270.7690 | 744301.3393 |
| 20 | 363.5883 | 257.8641 | 1000000 | 709219.8582 | 13 | 1397 | 260.2637 | 715819.6135 |
| 30 | 363.5883 | 256.9529 | 1000000 | 706713.7809 | 12 | 1403 | 259.1506 | 712758.3749 |
| 40 | 363.5883 | 252.5821 | 1000000 | 694240.8050 | 15 | 1426 | 255.2390 | 701543.4783 |
分区数不同
生产消息：
消息大小均为100byte、消息数：100W、节点数：3、副本数：1
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_3_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_6_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_9_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_12_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_15_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
消费消息：
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 17:36:08:761, 2020-10-21 17:36:09:683, 95.3674, 103.4354, 1000000, 1084598.6985, 11, 911, 104.6843, 1097694.8408
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_6_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 17:35:12:812, 2020-10-21 17:35:13:746, 95.3674, 102.1065, 1000000, 1070663.8116, 12, 922, 103.4354, 1084598.6985
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_9_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 17:37:00:262, 2020-10-21 17:37:01:155, 95.3674, 106.7944, 1000000, 1119820.8287, 13, 880, 108.3721, 1136363.6364
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_12_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 17:37:45:271, 2020-10-21 17:37:46:176, 95.3674, 105.3784, 1000000, 1104972.3757, 12, 893, 106.7944, 1119820.8287
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_15_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 17:38:29:756, 2020-10-21 17:38:30:659, 95.3674, 105.6118, 1000000, 1107419.7121, 13, 890, 107.1544, 1123595.5056
结果：
| 分区数 | 消息总大小(MB) data.consumed.in.MB |
每秒传输数据大小(MB) MB.sec |
消费消息总数 data.consumed.in.nMsg |
每秒消费消息数 nMsg.sec |
均衡耗时(ms) rebalance.time.ms |
拉取耗时(ms) fetch.time.ms |
每秒拉取数据(MB) fetch.MB.sec |
每秒拉取消息数 fetch.nMsg.sec |
|---|---|---|---|---|---|---|---|---|
| 3 | 95.3674 | 103.4354 | 1000000 | 1084598.6985 | 11 | 911 | 104.6843 | 1097694.8408 |
| 6 | 95.3674 | 102.1065 | 1000000 | 1070663.8116 | 12 | 922 | 103.4354 | 1084598.6985 |
| 9 | 95.3674 | 106.7944 | 1000000 | 1119820.8287 | 13 | 880 | 108.3721 | 1136363.6364 |
| 12 | 95.3674 | 105.3784 | 1000000 | 1104972.3757 | 12 | 893 | 106.7944 | 1119820.8287 |
| 15 | 95.3674 | 105.6118 | 1000000 | 1107419.7121 | 13 | 890 | 107.1544 | 1123595.5056 |
消息大小不同
生产消息：
消息数：100W、节点数：3、副本数：1、分区数：3、消息大小分别为：10byte、100byte、1000byte
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_3_1 --throughput 100000000 --record-size 10 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_3_1 --throughput 100000000 --record-size 100 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_3_1 --throughput 100000000 --record-size 1000 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
消费消息：
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 17:54:21:921, 2020-10-21 17:54:22:634, 9.5367, 13.3755, 1000000, 1402524.5442, 13, 700, 13.6239, 1428571.4286
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 17:54:56:167, 2020-10-21 17:54:57:050, 95.3674, 108.0039, 1000000, 1132502.8313, 13, 870, 109.6177, 1149425.2874
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1048576 --topic test_producer_3_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 17:55:25:427, 2020-10-21 17:55:27:890, 953.6743, 387.2003, 1000000, 406008.9322, 19, 2444, 390.2104, 409165.3028
结果：
| 消息大小 (bytes) |
消息总大小(MB) data.consumed.in.MB |
每秒传输数据大小(MB) MB.sec |
消费消息总数 data.consumed.in.nMsg |
每秒消费消息数 nMsg.sec |
均衡耗时(ms) rebalance.time.ms |
拉取耗时(ms) fetch.time.ms |
每秒拉取数据(MB) fetch.MB.sec |
每秒拉取消息数 fetch.nMsg.sec |
|---|---|---|---|---|---|---|---|---|
| 10 | 9.5367 | 13.3755 | 1000000 | 1402524.5442 | 13 | 700 | 13.6239 | 1428571.4286 |
| 100 | 95.3674 | 108.0039 | 1000000 | 1132502.8313 | 13 | 870 | 109.6177 | 1149425.2874 |
| 1000 | 953.6743 | 387.2003 | 1000000 | 406008.9322 | 19 | 2444 | 390.2104 | 409165.3028 |
每次拉取消息大小不同
生产消息：
消息数：100W、每条消息：1000byte、节点数：3、副本数：1、分区数：3、每次请求拉取数据大小分别为：1KB、1MB(默认)、10MB
./kafka-producer-perf-test.sh --num-records 1000000 --topic test_producer_3_1 --throughput 100000000 --record-size 1000 --producer-props bootstrap.servers=172.24.29.165:9092 acks=1
消费消息：
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1024 --topic test_producer_3_1 --group test-group-1 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 18:11:25:471, 2020-10-21 18:11:39:242, 953.6743, 69.2524, 1000000, 72616.3677, 12, 13759, 69.3128, 72679.7006
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 1024000 --topic test_producer_3_1 --group test-group-2 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 18:11:51:403, 2020-10-21 18:11:53:772, 953.6743, 402.5641, 1000000, 422119.0376, 11, 2358, 404.4420, 424088.2103
./kafka-consumer-perf-test.sh --broker-list 172.24.29.163:9092,172.24.29.164:9092,172.24.29.165:9092 --messages 1000000 --fetch-size 10240000 --topic test_producer_3_1 --group test-group-4 --timeout 60000 --threads 10
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-21 18:15:11:039, 2020-10-21 18:15:13:808, 953.6743, 344.4111, 1000000, 361141.2062, 13, 2756, 346.0357, 362844.7025
结果：
| 每次拉取大小 | 消息总大小(MB) data.consumed.in.MB |
每秒传输数据大小(MB) MB.sec |
消费消息总数 data.consumed.in.nMsg |
每秒消费消息数 nMsg.sec |
均衡耗时(ms) rebalance.time.ms |
拉取耗时(ms) fetch.time.ms |
每秒拉取数据(MB) fetch.MB.sec |
每秒拉取消息数 fetch.nMsg.sec |
|---|---|---|---|---|---|---|---|---|
| 1KB | 95.3674 | 69.2524 | 1000000 | 72616.3677 | 12 | 13759 | 69.3128 | 72679.7006 |
| 1MB | 95.3674 | 402.5641 | 1000000 | 422119.0376 | 11 | 2358 | 404.4420 | 424088.2103 |
| 10MB | 95.3674 | 344.4111 | 1000000 | 361141.2062 | 13 | 2756 | 346.0357 | 362844.7025 |