Kafka producer Producer parameter settings and parameter tuning suggestions-kafka business environment actual combat kafka business environment actual combat series

Kafka producer Producer parameter settings and parameter tuning suggestions-kafka business environment combat

Kafka business environment combat series

This series of blogs extracts cases from the real business environment for summary and sharing, and provides tuning suggestions for kafka commercial applications and cluster environment capacity planning. Please continue to pay attention to this set of blogs. Copyright statement: The copyright of this set of kafka tuning series belongs to the author (Qin Kaixin), reprinting is prohibited, welcome to learn.

Author: Qin Kai new address: Shenzhen

• Kafka business environment combat-kafka production environment planning
• Kafka business environment combat-Kafka producer and consumer throughput test
• Kafka business environment combat-Kafka producer Producer parameter settings and parameter tuning suggestions

1. Producer core workflow

• The Producer first uses the user's main thread to encapsulate the message to be sent into an instance of the ProducerRecord class.
• After serialization, it is sent to Partioner. After Partioner determines the target partition, it is sent to a memory buffer in the Producer program.
• Another worker thread of the Producer (Sender thread) is responsible for extracting the prepared messages from the buffer in real time, encapsulating them into a batch, and sending them to the corresponding broker in a unified manner.

2. Main producer parameter settings

2.1 producer parameter acks settings (no data loss)

The number of responses to the producer request that the producer needs to confirm from the leader before the message is considered "committed." This parameter is used to control the persistence of the message. Currently, 3 values ​​are provided:

acks = 0: Indicates that the produce request returns immediately, without waiting for any confirmation from the leader. This scheme has the highest throughput rate, but it does not guarantee whether the message is actually sent successfully.

acks = -1: Indicates that the partition leader must wait for the message to be successfully written to all ISR replicas (synchronized replicas) before considering the produce request as successful. This solution provides the highest message durability guarantee, but theoretically the throughput rate is also the worst.

acks = 1: It means that the leader copy must answer this produce request and write a message to the local log, after which the produce request is considered successful. If the leader copy dies after responding to the request at this time, the message will be lost. This is such a solution, which provides a good durability guarantee and throughput.

Business environment recommendation:

If you want higher durability requirements and no data loss requirements, set acks = -1. In other cases set acks = 1

2.2 producer parameter buffer.memory settings (throughput)

This parameter is used to specify the buffer size used by the Producer to cache messages, in bytes, the default value is 33554432 and the total is 32M. Kafka uses an asynchronous message architecture. When the prducer starts, it first creates a memory buffer to store the message to be sent, and then a dedicated thread is responsible for reading the message from the buffer for actual sending.

Business environment recommendation:

• In the process of continuous message sending, when the buffer is filled, the producer immediately enters the blocking state until the free memory is released. This period of time cannot exceed the value set by max.blocks.ms. Once exceeded, the producer will throw a TimeoutException. , Because Producer is thread-safe, if you keep reporting TimeoutException, you need to consider increasing buffer.memory.
• When users use multiple threads to share the Kafka producer, it is easy to fill up buffer.memory.

2.3 producer parameter compression.type setting (lZ4)

The producer compressor currently supports none (no compression), gzip, snappy and lz4.

Business environment recommendation:

Based on the company's Internet of Things platform, the current lz4 has the best effect. Of course, in August 2016, FaceBook open sourced Ztandard. Official website test: Ztandard compression ratio is 2.8, snappy is 2.091, LZ4 is 2.101.

2.4 producer parameter retries setting (note that the message is out of order, EOS)

The number of producer retries is set. When retrying, the producer will resend the previously failed message due to transient reasons. Reasons for transient failure may include: metadata information failure, insufficient number of copies, timeout, displacement out of bounds or unknown partitions, etc. If retries> 0 is set, then the producer will try to retry in these cases.

Business environment recommendation:

• The producer has another parameter: max.in.flight.requests.per.connection. If you set this parameter to about 1, then setting retries may cause out-of-order messages.
• Kafka version 0.11.1.0 already supports "one-time precise semantics", so message retry will not cause repeated message sending.

2.5 producer parameter batch.size setting (throughput and latency performance)

Producers are all sent in batches, so the choice of batch size is critical to producer performance. The producer will encapsulate multiple messages sent to the same partition into a batch. When the batch is full, the producer will send the messages out. But it does not necessarily wait until it is full, which is related to another parameter linger.ms. The default value is 16K, and the total is 16384.

Business environment recommendation:

• The smaller the batch, the lower the throughput of the producer, and the larger the throughput.
•  

2.6 producer parameter linger.ms setting (throughput and latency performance)

The producer sends it according to the batch, but it also depends on the value of linger.ms. The default is 0, which means no stay. In this case, some batches may not contain enough produce requests before being sent out, resulting in a large number of small batches, which puts great pressure on the network IO.

Business environment recommendation:

• In order to reduce network IO, improve the overall TPS. Assuming that linger.ms=5 is set, it means that the producer request may be sent with a delay of 5ms.

2.7 producer parameter max.in.flight.requests.per.connection settings (throughput and latency performance)

The maximum number of producer IO threads that can send unanswered produce requests on a single Socket connection. Increasing this value should increase the throughput of IO threads, thereby improving the performance of the producer as a whole. However, as mentioned before, if the retry mechanism is enabled, setting this parameter greater than 1 may cause the message to be out of order.

Business environment recommendation:

• The default value of 5 is a good starting point. If you find that the producer's bottleneck is in the IO thread, and the load on each broker is not high, you can try to increase the value appropriately.
• Increasing this parameter too much will cause the overall memory burden of the producer, and may also cause unnecessary lock competition, which will reduce TPS.