Kafka defaults to a single maximum size of 1M for the size of the message body. However, in our application scenario, there is often a message larger than 1M. If kafka is not configured, it will appear that the producer cannot push the message to kafka or the consumer cannot To consume the data in kafka, we need to configure the following for kafka:
1. Add in the consumer.properties configuration file: fetch.message.max.bytes : the number of message bytes fetched for each topic partition in each fetch request. Must be greater than or equal to message.max.bytes
2. Add in the producer.properties configuration file: max.request.size : The maximum size of the request is bytes. To be less than message.max.bytes
3. Add in the server.properties configuration file:
message.max.bytes : maximum record batch size allowed
replica.fetch.max.bytes : The number of message bytes each partition tries to fetch. Must be greater than or equal to message.max.bytes