Kafka stress testing can see where bottlenecks occur (CPU, memory, network IO). Generally, network IO reaches the bottleneck.
Create a test topic
kafka-topics --create --partitions 12 --replication-factor 1 --zookeeper node3:2181 --topic test
Producer stress test :
sh /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/kafka-producer-perf-test --topic test --record-size 2048 --num-records 1000000 --throughput -1 --producer-props bootstrap.servers=node3:9092,node4:9092,node5:9092
Explanation :
● record-size is how big a message is, the unit is bytes
● num-records is how many messages are sent in total
● throughput is how many messages per second, set to -1, which means no current limit, and the producer can be measured maximum throughput
Consumer Stress Test :
/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/kafka/bin/kafka-consumer-perf-test.sh \
--broker-list node3:9092,node4:9092,node5:9092 \
--topic test --threads 1 --fetch-size 10000 --messages 1000000
Description :
● broker-list broker node list
● fetch-size The size of the data for each fetch
● messages The total number of messages consumed
If the Kafka cluster cannot meet the performance requirements, partition expansion can be done to increase parallelism to improve throughput.
Kafka partition expansion :
and increase the parallelism of the downstream flink program, the number of kafka partitions = flink parallelism
sh /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/bin/kafka-topics --partitions 24 --alter --zookeeper node3:2181,node4:2181,node5:2181 --topic test
Kafka tuning
CM page kafka configuration tuning
num.network.threads=cpu核数+1 #网络通信处理线程数，默认值3
num.io.threads=cpu核数*2 #io磁盘io处理线程数，默认值8
replica.lag.time.max.ms= 30s # 如果网络不好,或者kafka集群压力较大,会出现副本丢失,然后会频繁复制副本,导致集群压力更大,此时可以调大该参数，默认值10s
broker_max_heap_size #堆内存大小，默认4G
batch.size=1M #默认16KB
buffer.memory=64M #默认32M
Producer side
compression.type=snappy #Data compression, no compression by default. Configuring a suitable compression algorithm can greatly reduce network pressure and Broker storage pressure.
Consumer side
//Increase poll quantity, default is 500 pull quantity
//fetch.max.bytes The default size of each batch of fetch is 50m
//Pull cycle
ZooKeeper tunes
the Java stack size (bytes) of ZooKeeper Server, 8G
maxClientCnxns (maximum number of client connections), default 1024, changed to 2048