In the actual Kafka development, we will find that whether it is a producer or a consumer, a Properties object needs to be constructed with many parameters set in it.
There are many commonly used parameter configurations in this code. When using it online, we have to determine the specific values of these configurations according to the actual data volume and data size. Let's pick out the more important parameters to analyze in detail.
Properties props = new Properties();
//集群地址，多个服务器用"，"分隔
props.put("bootstrap.servers", "192.168.72.141:9092,192.168.72.142:9092,192.168.72.143:9092");
//重新发送消息次数，到达次数返回错误
props.put("retries", 0);
//Producer会尝试去把发往同一个Partition的多个Requests进行合并，batch.size指明了一次Batch合并后Requests总大小的上限。如果这个值设置的太小，可能会导致所有的Request都不进行Batch。
props.put("batch.size", 163840);
//Producer默认会把两次发送时间间隔内收集到的所有Requests进行一次聚合然后再发送，以此提高吞吐量，而linger.ms则更进一步，这个参数为每次发送增加一些delay，以此来聚合更多的Message。
props.put("linger.ms", 1);
//在Producer端用来存放尚未发送出去的Message的缓冲区大小
props.put("buffer.memory", 33554432);
//key、value的序列化，此处以字符串为例，使用kafka已有的序列化类
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
//props.put("partitioner.class", "com.kafka.demo.Partitioner");//分区操作，此处未写
props.put("acks", "1");
props.put("request.timeout.ms", "60000");
props.put("compression.type","lz4");
1. Producer parameter acks setting
The number of responses to the produce request that the producer needs to confirm from the leader before the message is considered "committed". This parameter is used to control the persistence of the message. Currently, 3 values are provided:
- acks = 0: Indicates that the produce request returns immediately, without waiting for any confirmation from the leader. This scheme has the highest throughput rate, but it does not guarantee whether the message is actually sent successfully.
- acks = -1: Indicates that the partition leader must wait for the message to be successfully written to all ISR replicas (synchronized replicas) before considering the produce request as successful. This solution provides the highest message durability guarantee, but theoretically the throughput rate is also the worst.
- acks = 1: It means that the leader copy must answer this produce request and write a message to the local log, after which the produce request is considered successful. If the leader copy dies after responding to the request, the message will be lost. This program provides a good durability guarantee and throughput.
Recommended configuration:
If you want higher durability requirements and no data loss requirements, set acks = -1. In other cases, set acks = 1.
2. Producer parameter buffer.memory setting (throughput)
This parameter is used to specify the size of the buffer used by the Producer to cache messages, in bytes, the default value is 33554432 and the total is 32M. Kafka uses an asynchronous message architecture. When the prducer starts, it first creates a memory buffer to save the message to be sent, and then a dedicated thread is responsible for reading the message from the buffer for actual sending.
In the process of sending messages continuously, when the buffer is filled, the producer immediately enters the blocking state until the free memory is released. This period of time cannot exceed the value set by max.blocks.ms. Once exceeded, the producer will throw a TimeoutException. , Because Producer is thread-safe, if you keep reporting TimeoutException, you need to consider increasing buffer.memory.
When users use multiple threads to share the Kafka producer, it is easy to fill up buffer.memory.
3. The producer parameter compression.type setting
The producer compressor currently supports none (no compression), gzip, snappy and lz4.
In August 2016, FaceBook open sourced Ztandard. Official website test: Ztandard compression ratio is 2.8, snappy is 2.091, LZ4 is 2.101
4. Producer parameter retries setting
The number of producer retries is set. When retrying, the producer will resend the previously failed message due to transient reasons. Reasons for transient failure may include: metadata information failure, insufficient number of copies, timeout, displacement out of bounds or unknown partitions, etc. If retries> 0 is set, then the producer will try to retry in these cases.
5. The producer parameter batch.size setting
Producers are all sent in batches, so the choice of batch size is critical to producer performance. The producer will encapsulate multiple messages sent to the same partition into a batch. When the batch is full, the producer will send the messages out. But it does not necessarily wait until it is full, which is related to another parameter linger.ms. The default value is 16K, and the total is 16384.
6, producer parameter linger.ms setting
The producer sends it according to the batch, but it also depends on the value of linger.ms. The default is 0, which means no stay. In this case, some batches may not contain enough produce requests before being sent out, resulting in a large number of small batches, which puts great pressure on the network IO.
Recommended configuration:
In order to reduce network IO, improve the overall TPS. Assuming that linger.ms=5 is set, it means that the producer request may be sent with a delay of 5ms.