环境
- spring boot2
- spring cloud
- spring-kafka
- kakfa 2.2.0
场景
程序调用spring-kakfa
内置的kafkaTemplate
进行发送消息,但是消息实体过大,超过默认配置,导致消息发送失败。报错提示如下:
The message is 2044510 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.
原因
-
kakfa配置的消息大小(如
max.request.size
)偏小,导致入库失败; -
spring-kakfa的默认配置只有1M,导致报错。spring-kakfa在发送消息的时候,会先判断配置是否符合要求,符合再发送,否则就不发送。在这个过程,即使kakfa配置是偏大的,也会导致报错。通过查看类
org.apache.kafka.clients.producer.ProducerConfig
的说明可以得到,是先比较本地配置,如果不满足就不发送。查看注释如下:The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size. Note that the server has its own cap on record batch size which may be different from this.
解决措施
kakfa配置
-
server.properties
中添加message.max.bytes=5242880 # 每个分区试图获取的消息字节数。要大于等于message.max.bytes replica.fetch.max.bytes=6291456
-
producer.properties
中添加# 请求的最大大小为字节。要小于 message.max.bytes max.request.size = 5242880
-
consumer.properties
中添加# 每个提取请求中为每个主题分区提取的消息字节数。要大于等于message.max.bytes fetch.message.max.bytes=6291456
-
重启kakfa
# 关闭kakfa sh kafka-server-stop.sh # 启动 kakfa nohup sh kafka-server-start.sh ../config/server.properties &
spring-boot 配置修改
在配置文件加上以上配置:
spring.kafka.producer.properties.max.request.size=5242880