【Kafka】Message Queue Kafka Advanced

Kafka partition mechanism

Producer Partition Write Policy

The producer writes the message to the topic, and Kafka will distribute the data to different partitions according to different strategies.

• Round robin partition strategy
• random partition strategy
• Allocation strategy by key partition
• custom partition strategy

polling strategy

  The default strategy, which is also the most used strategy, can ensure that all messages are evenly distributed to one partition to the greatest extent. If the key is null when producing messages, the round robin algorithm is used to evenly distribute the partitions.

random strategy (not used)

  Random strategy, which randomly distributes messages to each partition every time. In earlier versions, the default partition strategy is random strategy, which is also to write messages to each partition in a balanced manner. But the follow-up polling strategy performs better, so the random strategy is rarely used.

Allocation strategy by key

  According to the key allocation strategy, there may be [data skew], for example: a key contains a large amount of data, because the key value is the same, all the data will be allocated to a partition, resulting in the number of messages in this partition is much larger than other partitions.

out of order problem

  Polling strategy and random strategy will cause a problem. The data produced in Kafka is stored out of order.
  Partitioning by key can achieve orderly storage of data to a certain extent—that is, local order, but this may lead to data skew, so in the actual production environment, a trade-off should be made based on the actual situation.

custom partition strategy

Create a custom partitioner:

public class KeyWithRandomPartitioner implements Partitioner {
    
    
    private Random r;

    @Override
    public void configure(Map<String, ?> configs) {
    
    
        r = new Random();
    }

    @Override
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
    
    
        //cluster.partitionCountForTopic 表示获取指定topic的分区数量
        //r.nextInt(1000) 表示从随机数生成器 r 中随机生成一个小于1000的整数，其中参数1000指定了生成的随机数的范围，即生成的随机数是0到999之间的整数。
        //在这段代码中，生成的随机数将被用于计算消息所在的分区编号。由于模运算 % cluster.partitionCountForTopic(topic) 的结果必须小于分区数量，因此这里对1000取模的目的是将随机数的范围缩小到分区数量内，以确保不会选择到超出范围的分区编号。
        return r.nextInt(1000) % cluster.partitionCountForTopic(topic);
    }

    @Override
    public void close() {
    
    }
}

In the Kafka producer configuration, customize the class name of the custom partitioner:

props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, KeyWithRandomPartitioner.class.getName());

Consumer Group Rebalance Mechanism

  Rebalance in Kafka is called rebalancing. It is a mechanism in Kafka to ensure that all consumers under the Consumer group reach a consensus and allocate each partition of the subscribed topic.

The timing of Rebalance triggering is:

• The number of consumers in the consumer group changes. For example: a new consumer is added to the consumer group, or a consumer stops.
  

• If the number of subscribed topics changes, consumers can subscribe to multiple topics. Suppose the current consumer group has subscribed to three topics, but one topic is suddenly deleted. At this time, rebalancing also needs to occur.

• 

• 
  
  Adverse effects of Rebalance when the number of subscribed topic partitions changes :

• When Rebalance occurs, all consumers under the consumer group will coordinate and participate together, and Kafka uses the allocation strategy to achieve the fairest allocation possible.

• The Rebalance process will have a very serious impact on the consumer group. During the Rebalance process, all consumers will stop working until the Rebalance is completed.

Consumer Partition Allocation Strategy

• Range allocation strategy
• RoundRobin polling strategy
• Stricky sticky allocation strategy

Range allocation strategy

Range distribution strategy is Kafka's default distribution strategy, which can ensure that the number of partitions consumed by each consumer is balanced.

Note : Range allocation strategy is for each Topic .

Configure the consumer partition.assignment.strategyas org.apache.kafka.clients.consumer.RangeAssignor:

props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.RangeAssignor");

Algorithm formula :

n = number of partitions / number of consumers

m = number of partitions % number of consumers

The first m consumers consume n+1

The remaining consumers consume n

RoundRobin polling strategy

The RoundRobinAssignor polling strategy is to sort the partitions of all consumers in the consumer group and all topics   subscribed by the consumers in lexicographical order (sort the hashcode of the topic and the partition), and then assign the partitions to each consumer.

Configure the consumer partition.assignment.strategyas org.apache.kafka.clients.consumer.RoundRobinAssignor:

props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.RoundRobinAssignor");

Stricky sticky allocation strategy

  Starting with Kafka 0.11.x, this type of allocation strategy is introduced. Main purpose: Partition distribution is as even as possible.
  When rebalance occurs, the allocation of partitions remains the same as the last allocation as much as possible. When no rebalance occurs, the Striky sticky allocation strategy is similar to the RoundRobin allocation strategy.

If consumer2 crashes above, rebalance needs to be performed at this time. If it is Range allocation and round-robin allocation, both will be re-allocated. For example,

it can be found that most of the partitions originally consumed by consumer0 and consumer1 have changed. Next, let's look at the sticky allocation strategy.

  The Striky sticky allocation strategy retains the allocation results before rebalance. In this way, the two partitions originally responsible for consumer2 are evenly distributed to consumer0 and consumer1. This can significantly reduce the waste of system resources.
  For example: consumer0 and consumer1 were consuming certain partitions before, but due to rebalance, consumer0 and consumer1 need to re-consume the partitions they were processing before, resulting in unnecessary system overhead. (For example: a transaction must be canceled if it is in progress)

Kafka replication mechanism

  The purpose of the copy is redundant backup. When the partition data on a Broker is lost, the data can still be guaranteed to be available. Because the replicas on other Brokers are available.

Producer's ACKs parameter

What is more relevant to the copy is the acks parameter configured by the producer. The acks parameter indicates the strictness of   writing to the copy when the producer produces messages . It determines how producers make trade-offs between performance and reliability .

// 配置ACKs参数
props.put("acks", "all");

acks is configured as 0

acks = 0: The producer only writes, regardless of whether the write is successful or not, data may be lost. Performance is top notch.

ACK is 0, benchmark:

bin/kafka-producer-perf-test.sh --topic benchmark --num-records 5000000 --throughput -1 --record-size 1000 --producer-props bootstrap.servers=10.211.55.8:9092 acks=0

acks is configured as 1

The producer will wait until the leader partition is successfully written, return success, and then send the next one, with medium performance.

acks is configured as -1 or all

Make sure that the message is written to the leader partition and the message is written to the corresponding replica successfully, and then send the next one. The performance is the worst.

Benchmark results:

index	Single partition single copy (ack=0)	Single partition single copy (ack=1)	Single partition single copy (ack=-1/all)
throughput	47359.248314 records/sec 4.7W records per second	40763.417279 records/sec 4W records per second	540.5 /s 7.3W tone record per second
Throughput rate	45.17 MB/sec about 45MB data per second	38.88 MB/sec about 89MB data per second	0.52 MB/sec
average latency	686.49 ms avg latency	799.67 ms avg latency	120281.8 ms
maximum delay time	1444.00 ms max latency	1961.00 ms max latency	1884.00 ms

  Select the ack mechanism according to the business situation, which requires the highest performance, and the loss of some data has little impact, so you can choose 0/1. If it is required that the data must not be lost, it must be configured as -1/all.

  There are the concepts of leader and follower in the partition. In order to ensure that the data consumed by consumers is consistent, only the leader of the partition can read and write messages. What the follower does is to synchronize data and backup.

High Level API and Low Level API

High Level APIs

• Consuming Kafka messages is easy to implement and relatively simple to write.
• There is no need to execute to manage the offset, and it is managed directly through ZK; there is no need to manage partitions and copies, and it is managed by Kafka.
• Consumers will automatically continue to obtain data according to the last offset saved in ZK.
• In ZK, different consumer groups (groups) record different offsets for the same topic, so that different programs read the same topic without being affected by the offset.
• Disadvantages of advanced API: cannot control offset, for example: want to read from a specified location; cannot finely control partitions, replicas, ZK, etc.

// 消费者程序：从test主题中消费数据
public class ConsumerTest {
    
    
    public static void main(String[] args) {
    
    
        // 1. 创建Kafka消费者配置
        Properties props = new Properties();
        props.setProperty("bootstrap.servers", "192.168.88.100:9092");
        props.setProperty("group.id", "test");
        props.setProperty("enable.auto.commit", "true");
        props.setProperty("auto.commit.interval.ms", "1000");
        props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        // 2. 创建Kafka消费者
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

        // 3. 订阅要消费的主题
        consumer.subscribe(Arrays.asList("test"));

        // 4. 使用一个while循环，不断从Kafka的topic中拉取消息
        while (true) {
    
    
            // 定义100毫秒超时
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
            for (ConsumerRecord<String, String> record : records)
                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
        }
    }
}

Low Level API

  By using the low-level API, we can control the offset by ourselves, and we can read from wherever we want.
  Moreover, you can control the connection partition yourself, and customize the load balancing for the partition. Moreover, before the offset was automatically saved in ZK, using the low-level API, you can store the offset without using ZK, and you can store the offset yourself.
  For example: stored in a file, MySQL, or in memory. However, the low-level API is more complicated. It needs to execute the control offset, which partition to connect to, and find the leader of the partition.

Example manual consumption of partitioned data:

  Before the high-level (High Level) API, we let Kafka dynamically assign partitions to be consumed for the topic according to the consumers in the consumption group. But at some point, it is necessary to specify the partition to be consumed, for example: if a program saves the data of a specified partition to external storage, such as Redis, MySQL, then when saving the data, only the specified partition needs to be consumed Partition the data. If a program is highly available, it will automatically restart when the program fails (for example: Flink, Spark program). In this case, the program will restart consuming data from the specified partition.

No longer use the previous subscribe method to subscribe to the topic, but use the assign method to specify the message you want to consume:

String topic = "test";
TopicPartition partition0 = new TopicPartition(topic, 0);
TopicPartition partition1 = new TopicPartition(topic, 1);
consumer.assign(Arrays.asList(partition0, partition1));

Once the partition is specified, the poll method can be called in a loop to consume messages just like the previous example.

Note :

When manually managing consumption partitions, even if the GroupID is the same, Kafka's group coordinator will no longer work.

There will also no longer be automatic partition reallocation if a consumer fails.

Monitoring tool Kafka-eagle

  In development work, when the business premise is not complicated, you can use Kafka commands to manage some clusters. But if the business becomes complicated, for example, it is necessary to add group and topic partitions, it is inconvenient to use the command line at this time. At this time, if a visual tool is used to help complete the daily management work, it will greatly improve the performance of Kafka. The efficiency of cluster management, and the use of tools to monitor consumer consumption in Kafka.

  Kafka Eagle is an excellent open source and free Kafka cluster monitoring tool redeveloped by combining the characteristics of the current big data Kafka monitoring tools. It can easily monitor offset, lag changes, partition distribution, owner, etc. in the production environment. Official website address: https://www.kafka-eagle.org/

Install Kafka-Eagle

Open the Kafka JMX port:

  JMX (Java Management Extensions) is a framework for embedding management functions into applications. JMX is a set of standard agents and services. In fact, users can use these agents and services to implement management in any Java application. Many software provides JMX interface to realize some management and monitoring functions.

Before starting the Kafka script, add:
【Kafka】Message Queue Kafka Basics --> Writing Kafka one-click startup/shutdown scripts has been added

cd ${KAFKA_HOME}
export JMX_PORT=9988
nohup bin/kafka-server-start.sh config/server.properties &

Install Kafka-Eagle :

The mysql database needs to be prepared and created in advance ke. Install JDK and configure JAVA_HOME.

# 将kafka_eagle上传，并解压到 /export/server 目录中。
cd cd /export/software/
tar -xvzf kafka-eagle-bin-3.0.1.tar.gz -C ../server/

cd /export/server/kafka-eagle-bin-3.0.1/ 
tar -xvzf efak-web-3.0.1-bin.tar.gz 
cd /export/server/kafka-eagle-bin-3.0.1/efak-web-3.0.1

# 配置 kafka_eagle 环境变量。
vim /etc/profile
export KE_HOME=/export/server/kafka-eagle-bin-1.4.6/kafka-eagle-web-1.4.6
export PATH=$PATH:$KE_HOME/bin
source /etc/profile

Configure kafka_eagle. Use vi to open the conf directorysystem-config.properties

vim conf/system-config.properties
# 修改第4行，配置kafka集群别名
kafka.eagle.zk.cluster.alias=cluster1
# 修改第5行，配置ZK集群地址
cluster1.zk.list=node1.itcast.cn:2181,node2.itcast.cn:2181,node3.itcast.cn:2181
# 注释第6行
#cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181
# 修改第32行，打开图标统计
kafka.eagle.metrics.charts=true
kafka.eagle.metrics.retain=30

# 注释第69行，取消sqlite数据库连接配置
#kafka.eagle.driver=org.sqlite.JDBC
#kafka.eagle.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
#kafka.eagle.username=root
#kafka.eagle.password=www.kafka-eagle.org

# 修改第77行，开启mysql
kafka.eagle.driver=com.mysql.jdbc.Driver
kafka.eagle.url=jdbc:mysql://10.211.55.8:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=root
kafka.eagle.password=52809329

# 启动脚本`ke.sh`中配置JAVA_HOME：
cd /export/server/kafka-eagle-bin-3.0.1/efak-web-3.0.1/bin

vim ke.sh
# 在第24行添加JAVA_HOME环境配置
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-arm64

# 修改Kafka eagle可执行权限
cd /export/server/kafka-eagle-bin-3.0.1/efak-web-3.0.1/bin
chmod +x ke.sh

# 启动 kafka_eagle
./ke.sh start

#访问Kafka eagle，默认用户为admin，密码为：123456
http://10.211.55.8:8048/ke

Kafka metrics topic list in the Kafka eagle interface :
Click the List menu under Topic to display all topics in the current Kafka cluster.

index	significance
Brokers Spread	Broker utilization
Brokers Skew	Whether the partition is skewed
Brokers Leader Skew	Is there any inclination in the leader partition?

Kafka principle

Partition leader and follower

  In Kafka, each topic can be configured with multiple partitions and multiple copies.
  Each partition has a leader and zero or more followers. When creating a topic, Kafka will evenly distribute the leader of each partition to each broker.
  Normal use of kafka does not feel the existence of leader and follower. But in fact, all read and write operations are handled by the leader, and all followers copy the log data files of the leader. If the leader fails, the follower will be elected as the leader.

Summary :

• The leader in Kafka is responsible for processing read and write operations, while the follower is only responsible for the synchronization of replica data.
• If the leader fails, other followers will be re-elected as the leader
• Like a consumer, the follower pulls the data corresponding to the leader's partition and saves it in the log data file

Check which server the leader of a partition is in:

Example topic named test with 3 partitions and 3 replicas.

AR、ISR、OSR

  In the actual environment, the leader may have some failures, so Kafka will definitely elect a new leader.
  In Kafka, followers can be divided into three categories according to different states - AR, ISR, OSR.

• All replicas of a partition are called AR( Assigned Replicasallocated replicas)
• All replicas (including the leader replica) that are in sync with the leader replica to some extent ISR( In-Sync Replicasreplicas in sync)
• Composition of replicas (excluding leader replicas) due to follower replica synchronization lag OSR( Out-of-Sync Replias)
• AR = ISR + OSR
• Normally, all follower replicas should be in sync with the leader replica, ie AR = ISR, OSRthe set is empty.

In Kafka, (C) is the smallest LEO in the ISR queue.
A.LEO

B. ISR

C. HW

D. AR

A: LEO: represents the next entry in the current log file, referring to the maximum offset of each copy

B: ISR: Represents the copy synchronization queue

C: HW: refers to the largest offset that consumers can see, the smallest LEO in the ISR queue, and C is correct .

D: AR: stands for all replicas

Controller

  When Kafka starts, a Controller will be selected among all brokers. The previous leader and follower are for the partition, while the Controller is for the broker.
  Management tasks such as creating topics, adding partitions, and modifying the number of replicas are all done by the Controller. The election of the Kafka partition leader is also decided by the Controller.

Controller election

  When the Kafka cluster starts, each broker will try to register as a Controller (ZK temporary node) on ZooKeeper, but only one succeeds in the competition, and other brokers will register the monitor of the node. Once the status of the temporary node changes, corresponding processing can be performed. The Controller is also highly available. Once a broker crashes, other brokers will re-register as Controllers.

Find the Controller of the current Kafka cluster: Go to
the Tools menu of Kafka Tools, find ZooKeeper Brower, and check which broker is the Controller.

The Controller elects the partition leader:

• The leader election of all Partitions is decided by the Controller
• The Controller will directly notify the Broker that needs to respond to the change of the leader through RPC
• The Controller reads the ISR of the current partition, and as long as one Replica survives, it selects one of them as the leader; otherwise, it randomly selects this Replica as the leader
• If all Replicas of the partition are down, the new leader is -1

  Specifically, when the leader copy of a partition fails, the follower copy will find out and send a request to other broker nodes to apply to become the new leader of the partition. At the same time, each broker node will periodically send a heartbeat request to the controller node to report its current status and availability information. The controller node will select a healthy and available broker node as the new leader of the partition based on this information.

In the process of electing a new leader, the Controller node will refer to the following factors :

• Replica status: Only follower replicas in the ISR (in-sync replicas) list are eligible to become the new leader because their data has been synchronized with the leader.
• Copy location: The Controller node will choose the same or earlier location as the original leader copy as the location of the new leader to ensure minimal data loss.
• Replica health status: The Controller node will preferentially select a healthy and available broker node as the new leader to ensure high availability and quality of service.

  In short, the Controller node will consider multiple factors and select a broker node that is most suitable to become the new leader, so as to ensure the high availability and stability of the Kafka cluster.

Why can't the leader of the partition be elected by ZK?

  If the Kafka cluster has a lot of business, there will be many partitions. Assuming that a certain broker is down, many partitons will need to re-elect the leader. If Zookeeper is used to elect the leader, it will bring huge pressure to Zookeeper. Therefore, the leader election in Kafka cannot be implemented using ZK.

When the previously offline partition comes back online, a Leader election must be performed, and the election strategy is ( A ).
A. OfflinePartition Leader Election

B. Reassign Partition Leader Election

C. PreferredReplicaPartition Leader Election

D. ControlledShutdownPartition Leader Election

The election of the leader copy of the partition is completely transparent to the user, and it is completed independently by the Controller.

A: OfflinePartition Leader election: Leader election needs to be performed whenever a partition goes online. The so-called online partition may be the creation of a new partition, or it may be that the previously offline partition has come online again. Option A is correct.

B: ReassignPartition Leader election: When you manually run the kafka-reassign-partitions command, or call Admin's alterPartitionReassignments method to perform partition copy reassignment, this type of election may be triggered.

C: PreferredReplicaPartition Leader election: When you manually run the kafka-preferred-replica-election command, or automatically trigger the Preferred Leader election, this type of policy is activated.

D: ControlledShutdownPartition Leader election: When a Broker shuts down normally, all Leader copies on the Broker will go offline. Therefore, it is necessary to perform a corresponding Leader election for the affected partition.

The request sent by the Controller to the Broker does not include ( D ).

A. LeaderAndIsrRequest

B. StopReplicaRequest

C. UpdateMetadataRequest

D. ActiveControllerCount

Controller will send three types of requests to Broker, namely LeaderAndIsrRequest, StopReplicaRequest and UpdateMetadataRequest .
A: LeaderAndIsrRequest: Tell the Broker on which Broker the Leader copy of each partition is located and on which Brokers the copies in the ISR are located.

B: StopReplicaRequest: Tell the specified Broker to stop the replica object on it, and the request can even delete the underlying log data of the replica.

C: UpdateMetadataRequest: This request updates the metadata cache on the Broker.

D: ActiveControllerCount is a monitoring indicator on the Broker side , not a sent request, so option D is not included.

leader load balancing

Preferred Replica

  Kafka introduces a preferred-replicaconcept called , which means: the preferred Replica is
  in the ISR list, the first replica is the preferred-replica, and the broker stored in the first partition must be the preferred-replica.

Execute the following script to set the preferred-replica as the leader and evenly distribute the leaders of each partition.

./kafka-leader-election.sh --bootstrap-server node1.itcast.cn:9092 --topic 主题 --partition=1 --election-type preferred

Make sure the leader is load balanced among the brokers:

Kill a broker of the test topic, so that Kafka will reassign the leader. After Kafka reassigns the leader, start the Kafka process again. At this point: Observe the distribution of the leader of each partition of the test topic.

At this time, the leader distribution will be uneven, so the following script can be executed to redistribute the leader:

bin/kafka-leader-election.sh --bootstrap-server 10.211.55.8:9092 --topic test --partition=1 --election-type preferred
# Successfully completed leader election (PREFERRED) for partitions test-1

–partition: Specify the partition number that needs to reassign the leader:

Kafka production, consumption data workflow

Kafka data writing process

• The producer first finds the leader of the partition from the "/brokers/topics/topic name/partitions/partition name/state" node of zookeeper:

• The producer finds the ID in ZK to find the corresponding broker:

• The leader on the broker process writes the message to the local log.
• The follower pulls the message from the leader, writes it to the local log, and sends an ACK to the leader
• After the leader receives ACKs from all Replicas in the ISR, it returns ACKs to the producer.

Kafka data consumption process

Two modes of consumption

  Kafka adopts the pull model , and the consumer records the consumption status by himself, and each consumer pulls the messages of each partition independently and sequentially. Consumers can consume messages in any order. For example, consumers can reset to the old offset and reprocess messages that have been consumed before; or jump directly to the nearest location and start consuming from the current moment.

• Each consumer can obtain the partition to consume according to the allocation strategy (default RangeAssignor)
• Obtain the offset corresponding to the consumer (by default, the offset of the last consumption is obtained from ZK)
• Find the leader of the partition and pull the data
• Consumer submits offset
  

Kafka's data storage form

  A topic consists of multiple partitions. A partition (partition) consists of multiple segments (segments), and a segment (segment) consists of multiple files (log, index, timeindex).

storage log

  The data in Kafka is stored in /export/server/kafka_2.12-2.4.1/data(installed in the configuration file settings), the messages are stored in 主题名-分区IDthe folder with:, and the data folder contains the following content:

file name	illustrate
00000000000000000000.index	Index file, the search data according to the offset is operated through the index file
00000000000000000000.log	log data file
00000000000000000000.timeindex	time index
leader-epoch-checkpoint	Persist the LEO corresponding to each partition leader (log end offset, offset of the next message to be written in the log file)

• The file name of each log file is the starting offset, because the starting offset of each partition is 0, so the log files of the partition start 0000000000000000000.logwith

• The default maximum size of each log file is log.segment.bytes =1024102410241G

• In order to simplify finding messages based on offset, the Kafka log file name is designed as the starting offset

Test: create a new topic: test_10m, the maximum size of each log data file for this topic is 10M

bin/kafka-topics.sh --create --zookeeper 10.211.55.8 --topic test_10m --replication-factor 2 --partitions 3 --config segment.bytes=10485760

Use the previous producer program to produce data in the test_10m topic:

Write message: New messages are always written to the last log file, and if the file reaches the specified size (default: 1GB), it will be rolled to a new file.

read message

According to the offset, you first need to find the segment segment that stores the data (note: offset specifies the global offset of the partition )
and then find the segment segment offset relative to the file according to the global

partition offset: finally read the message according to the segment segment offset , in order to improve the query Efficiency, each file will maintain a corresponding range of memory, and a simple binary search is used when searching.

delete message

  In Kafka, messages are cleaned up periodically. To delete log files of one segment at a time, Kafka's log manager will determine which files can be deleted according to Kafka's configuration.

No message loss mechanism

• Broker data is not lost
• Producer data is not lost
• Consumer data is not lost

Broker data is not lost

  After the producer writes data through the leader of the partition, all followers in the ISR will copy the data from the leader. In this way, it can be ensured that even if the leader crashes, the data of other followers is still available.

Producer data is not lost

  When the producer connects to the leader to write data, it can use the ACK mechanism to ensure that the data has been successfully written. The ACK mechanism has three optional configurations:

• When configuring the ACK response requirement to be -1 - it means that all nodes have received the data (both the leader and the follower have received
  the data)

• When configuring the ACK response requirement to be 1 - indicates that the leader has received data

• When the configuration ACK impact requirement is 0 - the producer is only responsible for sending data, and does not care whether the data is lost (in this case, data loss
  may occur, but the performance is the best)

Producers can send data in two ways: synchronous and asynchronous:

• Synchronization: After sending a batch of data to kafka, wait for kafka to return the result
• Asynchronous: Send a batch of data to Kafka, just provide a callback function.

Note: If the broker does not give ack for a long time and the buffer is full, the developer can set whether to directly clear the data in the buffer.

Consumer data is not lost

When consumers consume data, as long as each consumer records the ofiset value, the data can be guaranteed not to be lost.

data backlog

  Kafka consumers consume data very fast, but if there are some external IOs or network congestion when processing Kafka messages, data backlogs (or data accumulation) in Kafka will result. If the data has been backlogged, the real-time performance of the data will be greatly affected.

Lag label

Solve the data backlog problem

When Kafka has a data backlog problem, first find the cause of the data backlog. Following are a few class scenarios where data backlogs occur in an enterprise.

Failed to write data into MySQL


Problem description :
One day, the operation and maintenance personnel found the developer and said that a data backlog occurred in a partition of a certain topic. This topic was very important, and users began to complain. The operation and maintenance was very nervous, so I quickly restarted the machine. After restarting, it still doesn't help.

Problem analysis :

The code for consuming this topic is relatively simple, mainly consuming topic data, and then making judgments and performing database operations. The operation and maintenance found the backlog topic through kafka-eagle, and found that a certain partition of the topic had a backlog of hundreds of thousands of messages.
Finally, by checking the logs, it is found that the offset of the consumption partition has not been submitted since the data was written to MySQL and an error was reported, so the data backlog is serious.

Problem description due to network delay consumption failure :

The system developed based on Kafka has been running smoothly for two months. Suddenly, one day, it is found that there is a backlog of messages in a certain topic, and there are about tens of thousands of messages that have not been consumed.

Problem analysis :

By checking the application log, it is found that there are a large number of consumption timeout failures. After finding out the reason, because the network was jittering that day, the consumer timeout configuration of Kafka was set to 50ms, and then the problem was solved after changing the consumption time to 500ms.

Data cleaning in Kafka (Log Deletion)

  Kafka's messages are stored on the disk. In order to control the space occupied by the disk, Kafka needs to constantly clean up some past messages. Each partition of Kafka has a lot of log files, which is also for the convenience of log cleaning. In Kafka, two log cleaning methods are provided:

• Log deletion (Log Deletion): directly delete the unqualified logs according to the specified policy. Log deletion is performed periodically in units of segments (segment logs).

• Log Compaction: Consolidate according to the key of the message. For those with the same key but different values, only the last version is kept.

In Kafka's broker or topic configuration:

configuration item	configuration value	illustrate
log.cleaner.enable	true (default)	Turn on the automatic log cleaning function
log.cleanup.policy	delete (default)	delete log
log.cleanup.policy	compaction	compressed log
log.cleanup.policy	delete,compact	Support deletion and compression at the same time

Scheduled log deletion tasks

  There will be a dedicated log deletion task in the Kafka log manager to periodically detect and delete log segment files that do not meet the retention conditions. This cycle can be configured through broker-side parameters. The default value is 300,000, which is 5 minutes log.retention.check.interval.ms.

  There are three retention strategies for current log segments:

• Time-Based Retention Policy

• Retention policy based on log size

• Retention policy based on log start offset

Time-Based Retention Policy

  The following three configurations can specify that if the messages in Kafka exceed the specified threshold, the log will be automatically cleaned up: log.retention.hours, log.retention.minutes, log.retention.ms.
  Among them, the priority is log.retention.ms $>$ log.retention.minutes $>$ log.retention.hours。

  By default, in the broker, the configuration is as follows: log.retention.hours=168. That is, the default log retention time is 168 hours, which is equivalent to 7 days.

  When deleting log segments: remove the log segments to be deleted from the log segment jump table maintained in the log file object, so as to ensure that no thread reads these log segments. Add the suffix ".deleted" to the log segment file (including the index file corresponding to the log segment)

  Kafka's background timing task will periodically delete these files with the suffix ".deleted". The delay execution time of this task can be file.delete.delay.msset by parameters. The default value is 60000, which is 1 minute.

Set topic to delete once every 5 seconds :

For the convenience of observation, set the size of the segment file to 1M

and set the deletion policy of the topic: key: retention.ms value: 5000

  try to add some data to the topic, wait for a while, and observe the deletion of the log. It can be seen that logs are periodically marked for deletion and then deleted.

Retention policy based on log size

  The log deletion task will check whether the size of the current log exceeds the set threshold to find the file collection of log segments that can be deleted. It can be configured through broker-side parameters log.retention.bytes, and the default value is -1, which means infinity. If this size is exceeded, the excess will be deleted automatically.

Note :
log.retention.bytesWhat is configured is the total size of the log file, not the size of a single log segment. A log file contains multiple log segments.

Retention policy based on log start offset

  Each segment log has its start offset, if the start offset is less than logStartOffset, then these log files will be marked for deletion.

Log Compression

  Log Compaction is a way to clean up obsolete data other than the default log deletion. It will only keep one version of the data corresponding to the same key.

• After the Log Compaction is executed, the offset will no longer be continuous, but the Segment can still be queried
• Before and after Log Compaction is executed, the offset of each message in the log segment remains unchanged. Log Compaction will generate a new Segment file
• Log Compaction is for the key. When using it, pay attention that the key of each message is not empty
• Based on Log Compaction, the latest update of the key can be retained, and the latest state of consumers can be restored based on Log Compaction

Regarding Kafka's statement about cleaning up expired data, the error is (B)

A. Cleanup strategies include deletion and compression

B. After the compression policy is enabled, only the data of the specified version of each key is kept.

C. log.cleanup.policy=delete enable delete policy

D. log.cleanup.policy=compact enables compaction policy.

B. After the compression policy is enabled, only the data of the last version of each key is kept

Kafka quota speed limit mechanism (Quotas)

  Producers and consumers produce/consume a large amount of data or generate requests at a very high speed, thus occupying all resources on the broker and causing network IO saturation. With quotas (Quotas) these problems can be avoided. Kafka supports quota management, so that it can limit the flow of produce&fetch operations of Producer and Consumer to prevent individual businesses from overwhelming the server.

Limit producer-side rate

Set the default value for all client ids. The following sets the TPS for all producer programs to no more than 1MB/s, that is, 1048576/s. The command is as follows:

bin/kafka-configs.sh --zookeeper node1.itcast.cn:2181 --alter --add-config 'producer_byte_rate=1048576' --entity-type clients --entity-default

Run a benchmark to observe the rate of production messages:

bin/kafka-producer-perf-test.sh --topic test --num-records 500000 --throughput -1 --record-size 1000 --producer-props bootstrap.servers=node1.itcast.cn:9092,node2.itcast.cn:9092,node3.itcast.cn:9092 acks=1

# 结果：50000 records sent, 1108.156028 records/sec (1.06 MB/sec)

Limit consumer end rate

  The speed limit for consumer is similar to that for producer, but the parameter name is different. To limit the speed of the specified topic, the following sets the topic speed for all consumer programs to not exceed 1MB/s, that is, 1048576/s. The command is as follows:

bin/kafka-configs.sh --zookeeper node1.itcast.cn:2181 --alter --add-config 'consumer_byte_rate=1048576' --entity-type clients --entity-default

Run the benchmark:

bin/kafka-consumer-perf-test.sh --broker-list node1.itcast.cn:9092,node2.itcast.cn:9092,node3.itcast.cn:9092 --topic test --fetch-size 1048576 --messages 500000

#结果为：MB.sec：1.0743

Cancel the Quota configuration of Kafka

Use the following command to delete Kafka's Quota configuration:

bin/kafka-configs.sh --zookeeper node1.itcast.cn:2181 --alter --delete-config 'producer_byte_rate' --entity-type clients --entity-default

bin/kafka-configs.sh --zookeeper node1.itcast.cn:2181 --alter --delete-config 'consumer_byte_rate' --entity-type clients --entity-default