spring-kafka sets the size of the sent message

    Recently, the blogger encountered a problem of Kafka message size limit in the project, and recorded the problem and solution here;

Project problem environment : springboot2.2.2 kafka cluster

Scenes

The program calls spring-kakfathe built-in kafkaTemplateto send a message, but the message entity is too large and exceeds the default configuration, causing the message to fail to be sent. The error message is as follows:

The message is 2044510 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.

The reason
is that the message size (such as max.request.size) configured by kakfa is too small, which leads to the failure of storage;

The default configuration of spring-kakfa is only 1M, resulting in an error. When spring-kakfa sends a message, it will first determine whether the configuration meets the requirements, and then send it, otherwise it will not be sent. In this process, even if the kakfa configuration is too large, an error will be reported. By looking at the description of the class org.apache.kafka.clients.producer.ProducerConfig, it can be obtained that the local configuration is compared first, and if it is not satisfied, it will not be sent. Check out the notes below:

The maximum size of a request in bytes. This setting will limit the number of record  batches the producer will send in a single request to avoid sending huge requests. 
This is also effectively a cap on the maximum record batch size. 
Note that the server has its own cap on record batch size which may be different from this.

The solution
is to add in kakfa configuration
server.properties

message.max.bytes=5242880
# 每个分区试图获取的消息字节数。要大于等于message.max.bytes
replica.fetch.max.bytes=6291456

Add in producer.properties

# 请求的最大大小为字节。要小于 message.max.bytes
max.request.size = 5242880

add in consumer.properties

# 每个提取请求中为每个主题分区提取的消息字节数。要大于等于message.max.bytes
fetch.message.max.bytes=6291456

restart kakfa

# 关闭kakfa
sh kafka-server-stop.sh
# 启动 kakfa
nohup sh kafka-server-start.sh ../config/server.properties &

spring-boot configuration modification
Add the above configuration to the configuration file:

spring.kafka.producer.properties.max.request.size=5242880

 

{{o.name}}

{{m.name}}