Kafka introductory study notes arrangement

---

This article organizes the learning notes for getting started with Kafka:

• Reference video: kafka one-hour introductory intensive course

• Reference documents: Geek Time: Kafka Core Technology and Practice and Kafka Definitive Guide

This article only includes the introduction to Kafka, and the follow-up will continue to sort out Kafka's advanced knowledge and analysis of the underlying principles.

Most of the pictures in this article are personal hand-painted supplements, and the drawing software uses: draw.io

Complete code Demo project warehouse link: https://gitee.com/DaHuYuXiXi/kafak-demo

---

What is Kafka

Kafka is an open source platform for real-time streaming data processing developed by Linkedin. It can also be said to be a message queue middleware with distributed, multi-partition, multi-copy, multi-producer and consumer.

The message engine system needs to set a specific transmission protocol, that is, the method used to transmit the message. The common methods are:

• peer-to-peer model
• Publish Subscribe Model

Kafka supports both messaging engine models.

One of the functions of the message engine is to "cut peaks and fill valleys" to make up for the inconsistent production and consumption speeds of upstream and downstream systems.

---

Features of Kafka

• High throughput, low latency: Kafka can process data at a speed of several million pieces per second. The data processing speed is mainly affected by the size of each piece of data, and the delay of data transmission can be as low as a few milliseconds. The reason why Kafka can have such high performance is as follows:

  • Using zero copy avoids data copying from kernel space to user space
  • Data is written to the disk sequentially to avoid the time-consuming addressing caused by random writing to the disk. At the same time, because of sequential disk writing, Kafka does not provide the ability to address and delete data
  • Compress data to reduce IO latency
  • Use the method of sending data in batches instead of sending them one by one

• Persistence, reliability: Kafka will persist the received data to the disk for storage, and there is a multi-copy backup mechanism, so the persistence and reliability of the data are guaranteed to a certain extent

• High availability and fault tolerance: kafka divides messages into multiple topics (Topic), each topic is composed of multiple partitions, each partition has multiple copies, and partition copies are distributed in different servers (Broker). Because of the distributed architecture of kafka, if a node in the cluster goes down, the entire pool can still provide services to the outside world normally.

---

Application Scenario

• In business logic processing, convert non-core business logic to asynchronous, improve request processing speed, and complete decoupling between applications at the same time
  

1. For an order request, there must be an order user before the payment operation can be made, and the timeliness of changes in inventory and points does not require high timeliness, and can be processed asynchronously.

2. In the synchronization model, each module is usually aggregated into a single application, and each modification needs to be repackaged and deployed, which is more suitable for small team development. The asynchronous model can split each module to form a separate microservice, independent code, independent deployment, and mutual access through message queue or RPC, which is suitable for the development of large and complex projects.

---

• For high-concurrency processing, there are usually two solutions :
  • Horizontal expansion, increase the number of application instances, that is, elastic expansion, and handle high concurrency requirements with load balancing, but server resources are limited, the more servers you need, money need more
  • Classify message processing according to timeliness:
    • Real-time synchronous processing of highly time-sensitive messages, such as the above order creation
    • For messages with low timeliness and high concurrency, the message queue is used to buffer them first, in order to reduce the bottleneck of back-end service processing. Pull messages for processing according to the processing capabilities of the back-end service, instead of hitting all the requests to the back-end service at once, thus destroying the back-end services, such as; inventory and points management.

---

• Improve performance through batch processing : Since the message queue has buffered data, it creates conditions for us to batch process messages. The operations of batch receiving, batch processing, and batch warehousing of data must be better than our one-by-one data processing operations. Higher performance.

---

• Data collection and analysis system with database as the core: Record the running status of each microservice in the system, collect relevant running indicators and save them in the database, and at the same time, the third-party system queries the database and displays it visually. Alarm, periodic aggregate calculation of each indicator data, such as: maximum value, minimum value, average value

question:

• The interaction is centered on the database, and the pressure on the DB is huge
• Due to the interaction centered on the database, the data delay is very large, because the database query statistics take time, and the application service may be down when we find the threshold alarm
• Database query cannot achieve real-time statistics

---

• Change to a stream data processing system with Kafka as the core :
  

After the collector collects the indicators, it sends them to the message queue:

• System management application, monitor the data topic, save the data to DB after receiving it
• Data monitoring application, after receiving the data, judge whether the data exceeds the threshold, and if it exceeds the threshold, send an alarm message
• The third-party system B directly monitors the specified topic of the message queue to obtain real-time data
• Count the maximum, minimum, and average values ​​of indicators every 5, 15, 30, and 60 minutes. Kafka provides relevant time window capabilities, which can be used for effective statistics. This is provided by stream processing systems such as Kafka and Flink. Streaming data statistics function, if you don’t know how to use it, you can also code it yourself, use related variable records, and update the variable value every time a message is received.

benefit:

• The message delay of Kafka as a message queue is very low, which can meet the real-time requirements

---

• Kafka Connect provided by Kafka can standardize the movement of various data from various data sources into Kafka, and provide a standardized Sink to move data into some kind of data storage or database, and it is available for common databases or big data application storage Very good support, such as: mysql, HDFS, etc.

• User behavior tracking : such as e-commerce shopping, when you open an e-commerce shopping platform, your login user information, login time and location and other information; when you browse products, the category, price, store and other information of the products you browse are all It can be passed to Kafka through Kafka messages, and through real-time streaming computing, product recommendations can be made to you according to your preferences.
• Metrics: For example, Kafka can be used to transmit server metrics monitoring data. When the server is abnormal, it can analyze the server monitoring metrics data in real time and generate an alarm accordingly.
• Streaming processing: Streaming data processing is relative to batch data processing. Data is processed in the form of data flow, focusing on the timeliness of data, rather than processing the data after it is stored in the database. This is usually referred to as called batch
• Current limiting and peak shaving: In the scenario where high concurrent user requests impact the application system, Kafka can effectively serve as a data buffer, avoiding a large number of concurrency directly facing service programs and databases, and avoiding excessive pressure on back-end programs and databases causing service Crash.

---

Kafka installation

Stand-alone deployment

---

Cluster deployment environment preparation

• Configure the java environment
• Configure the mapping relationship between ip and host name in the /etc/hosts file

The hostnamectl set-hostname hostname command can be used to set the hostname. After the modification, reopen a terminal to see that the modification takes effect, or use the hostname command to view the modified host name

• The /etc/hosts file is the file responsible for fast resolution of IP and domain name in the Linux system. This file contains the mapping relationship between IP address and host name, and also includes the alias of the host name. In the absence of DNS domain name server, the system All network programs on the Internet query this file to resolve the IP address corresponding to a certain host name, otherwise they need to use the DNS service to complete the resolution.
• The priority of the /etc/hosts file is higher than that of the DNS service program

The /etc/hosts file usually contains the following:

127.0.0.1       localhost       localhost.localdomain   localhost4      localhost4.localdomain4
172.23.57.29    dhy.com dhy

One mapping relationship per line, consisting of three parts: IP address, domain name or host name, host name

The difference between a host name and a domain name: The host name is usually used in the local area network. Through the hosts file, the host name is resolved to the corresponding IP. The domain name is usually used on the Internet, and DNS resolution is used. If the machine does not want to use the domain name resolution on DNS, you can Change the hosts file and add your own domain name resolution.

Suppose we have three servers, and add the following mapping relationship in the /etc/hosts files of the three servers:

192.168.110.92 kafka-0
192.168.110.93 kafka-1
192.168.110.94 kafka-2

At this time, if we are at 192.168.110.92 and want to remotely log in to 192.168.110.93 through ssh, we only need to access through the host name, and then enter the password to log in:

ssh root@kafka-1

• Create user – Example: create a new user called kafka
• Firewall-related ports are open (the cloud server needs to go to the web page to open)

---

• Password-free login settings between clusters

Asymmetric encryption:

• The issuer creates a secret key pair, the key kept by the issuer is called the private key, and the key publicly released to others is called the public key
• The public key encryption result can be decrypted by the private key; the private key encryption result can also be decrypted by the public key.

SSH password-free login schematic diagram:

• If host1 wants to log in to host2 without secret, then the key pair is issued by host1
• Let host2 trust the public key of host1, and host1 can log in to host2 without secret, so host1 needs to save a copy of its own public key on the host2 server, that is, copy the secret key

In the host that is logged in without secret by SSH, there is a file that stores the public key of the host to log in. The name is authorized_keys, and its location exists in the directory /登录用户根目录/.ssh:

If the current host has not been configured with any password-free login, this file does not exist by default

In the authorized_keys file, the identity information of other hosts that can log in to the local host is stored. If the key generated by the rsa algorithm is used, the storage format of the file is a set of strings beginning with ssh-rsa.

Specific steps:

• host environment

ip             主机名    用户   
192.168.110.92 kafka-0  kafka
192.168.110.93 kafka-1  kafka
192.168.110.94 kafka-2  kafka

• generate key pair

ssh-keygen -t rsa
//执行完毕后,会在/home/kafka/.ssh目录下看到如下两个文件,通常认为前者是公钥,后者是私钥:
id_rsa.pub
id_rsa

• key handling

//1.将公钥保存到authorized_keys文件中
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
//2.将公钥分发给kakfa-1、kakfa-2主机。按提示输入kafka登陆密码
ssh-copy-id -i ~/.ssh/id_rsa.pub -p22 kafka@kakfa-1;
ssh-copy-id -i ~/.ssh/id_rsa.pub -p22 kafka@kakfa-2;
//3.我们现在就可以通过kakfa-0主机免密登录kakfa-1和kakfa-2了

• Password-free login test: Log in to the kakfa-1 (or kakfa-2) host on the kakfa-0 host, and enter the following command without entering a password. You will find that the login host switch is as follows, and the login is completed without a password

ssh kafka@kakfa-1

Repeat the above steps on the kakfa-1 and kakfa-2 servers to fully realize the ssh password-free login between the three servers

---

Kafka 2.x cluster deployment

---

Kafka 3.x cluster deployment

In kafka 3.0, zk has been removed, and the kraft mechanism is used to realize the election of the controller master controller:

• 2.0: All nodes in a cluster are Brokers, and one of them is elected as the Controller controller. The controller saves cluster metadata information, such as: topic classification, consumption progress, etc., to zk for distributed distribution between nodes in the cluster interact
  

• 3.0: Assuming that a cluster consists of four Brokers, we artificially designate three of them as Controllers, elect one Controller from the three Controllers as the main controller, and the other two are standby. zk is no longer needed, and all cluster metadata is stored in Kafka's internal topics, which are managed by Kafka itself and no longer rely on zookeeper
  

• Before building a kafka cluster, we need to plan the role of the kafka instance

ip             主机名   角色                  node.id
192.168.110.92 kafka-0  broker,controller      1
192.168.110.93 kafka-1  broker,controller      2
192.168.110.94 kafka-2  broker,controller      3
192.168.110.94 kafka-3  broker                 4

Controller needs an odd number, which is the same as zk

• Download the Kafak installation package: download from the domestic Ali mirror address

Kafka 3.0 no longer supports JDK8, it is recommended to install JDK11 or JDK17

• Create a new kafka installation directory in the kafka user's home directory, and decompress the installation package to this directory

tar -zxvf kafka_2.12-3.1.2.tgz

• Create a new directory to save Kafka's persistent log data mkdir -p /home/kafka/data, and ensure that the user who installs Kafka has read and write permissions to the directory
• All installed kafka server instances open ports 9092 and 9093, and use this port as the communication port between controllers, which is similar to port 2181 of zk
• Modify the Kraft protocol configuration file: In kafka 3.0 version, the Kraft protocol is used instead of zk for cluster controller election, so it needs to be configured. The configuration file is in the kraft directory, which is different from the configuration file of kafka 2.0 version that relies on the zk installation method of
  
• The specific configuration parameters are as follows: (each node is configured)

node.id=1
process.roles=broker,controller
listeners=PLAINTEXT://kafka-0:9092,CONTROLLER://kafka-0:9093
advertised.listeners = PLAINTEXT://:9092
controller.quorum.voters=1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
log.dirs=/home/kafka/data/

• node.id: This will serve as the node ID in the cluster. For the broker.id in kafka2.0, it is just that in version 3.0, the kafka instance no longer only assumes the role of broker, but may also be the role of controller, so it is renamed as node node.
• process.roles: A node can act as broker or controller or both.
• listeners: The broker will use port 9092, and the kraft controller will use port 9093.
• advertised.listeners: The address leaked by kafka through the proxy. If it is used in the LAN, just configure PLAINTEXT://:9092.
• controller.quorum.voters: Specifies the voting node for the controller to control the election. All planning nodes whose process.roles includes the controller role must participate, namely: kafka-0, kafka-1, and kafka-2. Its configuration format is: node.id1@host1:9093, node.id2@host2:9093
• log.dirs: The log directory where kafka will store data.

• Format the storage directory (come to the kafka installation bin directory)

//1.生成一个唯一的集群ID（在一台kafka服务器上执行一次即可），这一个步骤是在安装kafka2.0版本的时候不存在的。
./kafka-storage.sh random-uuid
//2.使用生成的集群ID+配置文件格式化存储目录log.dirs,所以这一步确认配置及路径确实存在，并且kafka用户有访问权限（检查准备工作是否做对）。每一台主机服务器都要执行这个命令
kafka-storage.sh format \
-t 集群ID \
-c /home/kafka/kafka_2.12-3.1.2/config/kraft/server.properties
//3.格式化操作完成之后，你会发现在我们定义的log.dirs目录下多出一个meta.properties文件。meta.properties文件中存储了当前的kafka节点的id(node.id)，当前节点属于哪个集群（cluster.id）

• Start the cluster for testing

//依次在每台机器上执行下面这条启动命令
./kafka-server-start.sh /home/kafka/kafka_2.12-3.1.2/config/kraft/server.properties

• Create a topic for testing ( Note: The Kafka 3.0 version is different from the previous version to create a topic command, since it does not rely on zk startup, fill in the Broker address here )

./kafka-topics.sh  --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test1

The new version of Kafka no longer needs to rely on zookeeper to create topics, and no longer needs zookeeper to save metadata information

Kafka version 2.8 introduces a major improvement: the KRaft mode. This feature has been in the experimental stage. On October 3, 2022, Kafka 3.3.1 was released, officially declaring that the KRaft mode can be used in a production environment. In KRaft mode, all cluster metadata is stored in Kafka's internal topics, managed by Kafka itself, no longer dependent on zookeeper

Many old kafka versions only use –zookeeper ip:2181 to connect to zookeeper and then control the broker service to execute commands. Although this parameter is still supported in newer versions of kafka, it is not recommended because it is in the development roadmap of kafka Zookeeper will be eliminated gradually. Therefore, it is recommended that you use –bootstrap-server ip:9097 to connect to the service.

• Check the status of the topic

./kafka-topics.sh   --bootstrap-server localhost:9092  --describe --topic test1

• Test with producers and consumers

#消费者窗口监听主题中的消息
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1
#生产者窗口往主题中投递消息
./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test1

---

Listeners and internal and external networks

• Listener: Specify the listening name and port number of the local machine when the Broker instance starts, and use it on the server side

 listeners = listener_name://host_name:port,listener_name2://host_name2:port2
 例如:
  listeners = PLAINTEXT://:9092
  listeners = PLAINTEXT://192.168.1.10:9092
  listeners = PLAINTEXT://hostname:9092
  #监听所有网卡
  listeners = PLAINTEXT://0.0.0.0:9092

• The name and port of the listener must be unique, if the port is the same, there will be a conflict
• If host_name is empty, all network cards will be bound, that is to say, no matter which network card comes in, the request will be accepted and processed.
• listener_name is the listener name, the only value, it is not a security protocol, because the default 4 security protocols have been mapped:

格式为:
监听名称1:安全协议1,监听名称2:安全协议2
默认四个映射:
PLAINTEXT => PLAINTEXT 不需要授权,非加密通道
SSL => SSL 使用SSL加密通道
SASL_PLAINTEXT => SASL_PLAINTEXT 使用SASL认证非加密通道
SASL_SSL => SASL_SSL 使用SASL认证并且SSL加密通道

• advertised.listeners: access IP and port published externally, this information will be registered in zk storage (version 2.x) or kafka internal topic (version 3.x), for client use
  
• If avertised.listeners is not configured, the configuration of listeners is used by default
• In the figure below, Broker is configured through avertised.listeners, and what is exposed externally is the internal network IP or host name, which requires that the client must be on the same network segment as the Broker instance to access ((The figure below shows kafka 2.x version, will Metadata information is stored in zk))
  
• Broker instances can be accessed through the internal network or hostname
  
• The listeners are configured with multiple listeners, and the internal and external networks are split, so that the Brokers will directly connect to each other through the internal network, and the client can not only connect through the internal network (if it is in the internal network environment), but also connect through the external network.
  

example:

listeners=INTERNAL://:9092,EXTERNAL://0.0.0.0:9093
advertised.listeners=INTERNAL://kafka-0:9092,EXTERNAL://公网IP:9093
#设置监听器名称和安全协议之间的映射关系集合。
#注意:自定义了监听器,则必须要配置inter.broker.listener.name
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
#用于Broker之间通信的listener的名称,如果未设置，则listener名称由 security.inter.broker.protocol定义,默认为PLAINTEXT
inter.broker.listener.name=INTERNAL

For details, please refer to:
[Cloud Native] An article to understand listeners and advertised.listeners in Kafka and other communication configurations

---

KRaft model

Before version 2.8, Kafka strongly relied on zk, a distributed service coordination management tool. In version 2.8 of kafka, it began to try to remove zk from the service architecture, and the work was basically completed in version 3.0.

• What metadata information does zk save in kafka 2.x version?
  • In Kafak 2.x version, zk is responsible for saving the metadata information of the Kafka cluster operation, mainly including the operating status information and configuration information of some cluster nodes.

/admin ： 用于保存kafka集群管理相关的信息，如已经被删除的topic。
/brokers ： 用于保存当前集群的所有broker的id，和已经创建未被删除的topic
/cluster : 用于保存kafka集群的id，kafka的集群存在一个唯一的id及版本信息保存在这里
/config : 集群运行过程中的客户端、服务端、主题、用户等配置信息
/controller ： 用于保存kafka集群控制器组件的信息，如：版本号、控制器在哪个broker上、时间戳信息。
/controller_epoch ：用于记录controller选举的次数，每完成一次controller选举，该数据就加1。
/isr_change_notification ： ISR列表发生变更时候，发出的通知信息。
/latest_producer_id_block ：每个 Producer 在初始化时都会被分配一个唯一的 PID，pid开始结束范围以及申请结果保存在这里。
/log_dir_event_notification ：如果broker在向磁盘写入数据的时候出现异常，信息保存在这里。 controller监听到该目录的变化之后会进行相应的处理。

• What are the problems caused by Kafka's strong dependence on zk?

  • As an external service component, zk saves metadata information related to the Kafka cluster, which means that all operations need to be realized through a large number of network interactions, and the overhead of network IO reduces the performance of the cluster
  • Need to maintain the stability of the zk cluster
  • Too many zk directory nodes will affect performance. At the same time, each znode node has a data limit of 1M. As the Kafka cluster becomes larger, the data storage capacity of zk itself will also affect performance.

• Kafka version 2.8 introduces a major improvement: the KRaft mode. This feature has been in the experimental stage. On October 3, 2022, Kafka 3.3.1 was released, officially declaring that the KRaft mode can be used in a production environment. In KRaft mode, all cluster metadata is stored in Kafka's internal topics, which are managed by Kafka itself and no longer rely on zookeeper.
  

• In the service architecture of kafka2.0, the yellow icon represents the controller, and the black icon represents the broker. All brokers rely on an elected controller to control and manage them. The controller service instance is actually one of the three brokers, one of which is elected and given the role of controller.

• Under the kafka3.0 (KIP-500) service architecture, there are four brokers in the above figure, three of which are assigned the role of controller. One of the three controllers is elected as the master controller.

Zookeeper's distributed data service coordination capability is replaced by the Raft protocol in Kafka3.0, so that the election of the leader controller and the consistency of the partition copy are guaranteed

After Kafka removes ZK, it is easier to deploy and maintain, and at the same time, it is more convenient to monitor and implement, and its performance has also been greatly improved.

---

Docker installation

---

Topic management and cluster fault tolerance test

topic creation and status query

• Create Topics

# 创建名为topic的主题,该主题下只有一个分区,每个分区两个副本的topic
./kafka-topics.sh --create \
--bootstrap-server kafka-0:9092,kafka-1:9092,kafka-2:9092 \
--replication-factor 3 --partitions 1 \
--topic test2

bootstrap.servers is only used when the client starts (bootstrap) to have a connector that can be hot-started. Once the startup is complete, the client should be able to know the information of all nodes in the current cluster. When the cluster expands in the future, the client It can also automatically get the information of the new node in real time. Even if the bootstrap.servers hangs up, it should be able to run normally, unless the client restarts after the node hangs up.

• View topic detailed status

./kafka-topics.sh --bootstrap-server kafka-0:9092,kafka-1:9092,kafka-2:9092 --describe -topic test2

The format of the result returned by this command is as follows:

Topic: test2     TopicId: Sf8chBRzRoWs2oBSoXNsbQ PartitionCount: 1       ReplicationFactor: 3    Configs: cleanup.policy=delete,flush.ms=1000,segment.bytes=1073741824,flush.messages=10000
        Topic: test     Partition: 0    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3

• Topic: name test2
• PartitionCount: number of partitions 1
• ReplicationFactor: There are 3 partition copies in total, including 1 Leader partition copy and 2 Follower partition copies
• Partition: 0: This record corresponds to the partition number 0 (a partition number starts from 0), if there are multiple partitions: Partition: 1 、Partition: 2, and so on.
• Leader: 1: Indicates that the primary partition is on the node with broker.id=1
• Replicas: 1,2,3: Indicates that the three copies of the partition are on the nodes of broker.id=1\2\3
• Isr: 1,2,3: Indicates that all 3 partition copies are in the Isr set, and 2 Followers and 1 Leader are in sync.

---

fault tolerance test

We need to test the following:

• When all nodes in the cluster are normal, production and consumption are guaranteed to be normal

//1.在某个节点上生产者发送数据
./kafka-console-producer.sh --topic test --bootstrap-server kafka-0:9092,kafka-1:9092,kafka-2:9092
//2.在另一个节点上消费者接收数据
//参数--from-beginning的作用是使consumer从kafka最早的消息开始消费
./kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka-0:9092,kafka-1:9092,kafka-2:9092
//3.检查消息收发是否正常

The ability to send and receive data normally proves that all nodes in the cluster are in a normal state and production and consumption are normal. If you cannot normally check your own installation process configuration, whether the cluster is started, kafka service logs and other information.

• Stop some follower copies, production and consumption must still be normal
• When the leader copy is stopped and there are follower copies available, production and consumption must still be normal

If we stop the kafka process on the server where the Follwer replica broker.id=2 is located:

./kafka-server-stop.sh

At this time, check the partition status of test2 topic, and you will find that only partitions 1 and 2 are left in the ISR set:

Topic: test2     Partition: 0    Leader: 1       Replicas: 1,2,3 Isr: 1,2

At this time, if the producer continues to send data, there will be two possible results:

• The producer sends normally, and the consumer consumes normally
• The production method is sent normally, and the consumer cannot consume the data

Why?

• There is a special topic in kafka __consumer_offsets, which is used to save the progress offsets of production data and consumption data of other topics. This topic has 50 partitions. 核心问题默认情况下是它的分区副本因子是1，也就是说每个分区只有一个Leader副本.

We can see it with the following command:

./kafka-topics.sh --bootstrap-server kafka-0:9092,kafka-1:9092,kafka-2:9092 --describe -topic __consumer_offsets

Assume that the data production progress offset of topic test2 needs to be saved to __consumer_offsetspartition 2 of the topic, but we stop the server with broker.id=2, and there is only one leader in the partition without backup.

The producer data is normally written to the test2 topic, but the production progress offset of the topic needs to be updated __consumer_offsets. If it cannot be updated, the consumer cannot consume this data.

In order to achieve high availability of the cluster, the following things need to be done:

• The partition of the theme we built ourselves needs to have multiple copies, specify the parameter –replication-factor n when creating the theme
• __consumer_offsetsThere are also multiple copies, which can be specified in the configuration file:

# 创建topic时候，默认的分区数
num.partitions=3
# 允许默认创建Topic
auto.create.topics.enable=true
# __consumer_offsets的分区数设置（n>1,建议值3）
offsets.topic.replication.factor=n

The three parameters mentioned above take effect when the __consumer_offsets topic is initialized and created (when the cluster produces the first message), so even if the cluster is restarted, the number of partition copies of the __consumer_offsets topic is still 1 at the beginning

If __consumer_offsetsthe partition contains n=3 replicas, unless all replicas are broken, it can still work normally, so the more replicas, the higher the availability of the cluster, but the more resources consumed by data storage and data synchronization between replicas .

---

Dynamically modify the number of partitions

If we want to dynamically modify the number of partitions of a topic, here we take __consumer_offsetsthe topic as an example, first create a JSON file, such as topic.json:

{
    
    
    "version": 1, 
    "partitions": [
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 0, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 1, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 2, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 3, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 4, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 5, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 6, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 7, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 8, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 9, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 10, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 11, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 12, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 13, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 14, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 15, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 16, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 17, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 18, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 19, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 20, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 21, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 22, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 23, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 24, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 25, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 26, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 27, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 28, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 29, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 30, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 31, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 32, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 33, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 34, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 35, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 36, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 37, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 38, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 39, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 40, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 41, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 42, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 43, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 44, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 45, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 46, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 47, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 48, 
            "replicas": [1,2,3]
        },
        {
    
    
            "topic": "__consumer_offsets", 
            "partition": 49, 
            "replicas": [1,2,3]
        }
    ]
}

There are a total of 50 partitions, and three copies of each partition are distributed on three servers with broker.id=1,2,3 , which is the meaning of the above json file. Put the file into the explain directory of the kafka installation directory.

Then execute the following command:

./kafka-reassign-partitions.sh \
--bootstrap-server kafka-0:9092,kafka-1:9092,kafka-2:9092 \
--reassignment-json-file ./topic.json --execute

After execution, we can execute the following command to verify:

./kafka-reassign-partitions.sh \
--bootstrap-server kafka-0:9092,kafka-1:9092,kafka-2:9092 \
--reassignment-json-file ./topic.json --verify

If the result of the verification is "is complete" for each partition, it means that the distribution of partition copies is completed correctly.

After configuring the __consumer_offsets topic above:

• For three Broker nodes, even if one node or two nodes are stopped, it will not affect the normal reception of producer messages by the cluster, nor will it affect consumers to consume. Even if the partition leader copy is stopped, if there is a follower partition copy alive, the leader will be re-elected. In addition to wasting a certain amount of election time, the cluster as a whole is still available.

---

Basic concepts in Kafka

• Topic: In Kafka, the object of publishing and subscribing is a topic. You can create a dedicated topic for each business, each application or even each type of data.

Producers continuously send messages to topics, consumers continuously pull messages from topics for consumption, and both producers and consumers can send or pull messages to one or more topics at the same time:

• Broker (message broker): A Broker and a Kafka service instance. The Kafka cluster is composed of multiple Brokers. The Broker is responsible for receiving and processing client requests and persisting messages.
  
  Usually, different Brokers in the Kafka cluster are distributed and run on different machines to prevent the downtime of one machine from causing the entire cluster to be unavailable.

• Relication (copy): Kafka defines two types of copies: leader copy and follower copy. The former provides read and write services to the outside world, while the latter does not provide read and write services to the outside world. It is only responsible for synchronizing the data of the leader copy. Additional storage to ensure that data is not lost.

Note: Producers always write messages to the leader replica, and consumers always read messages from the leader replica. As for the follower copy, it only does one thing: send a request to the leader copy, requesting the leader to send it the latest production message, so that it can keep in sync with the leader's data.

The follower copy is only responsible for synchronizing the data of the leader copy, and the failover is achieved through the election of the follower copy.

• Partitioning (partition): In the case of a large amount of data, a single leader copy will accumulate too much data, so that a single Broker machine cannot accommodate it, then you should consider dividing the data into multiple copies to keep in different On the Broker, this mechanism is called partitioning, which can be compared to: the fragmentation mechanism in ES and Redis.

A partition is a real physical queue data structure used to store data, occupying resources such as system memory and disk data storage.

The partition mechanism in Kafka is to divide each topic into multiple partitions. Each partition is a set of ordered message logs. The number of partitions included in a Topic depends on the throughput capability requirements of commodity processing under the topic.

Each message produced by the producer will be sent to one of the partitions. The specific partition to be sent to is determined by the specific message routing strategy. The default is the polling strategy.

Kafka's partition numbers start from 0.

Replicas are defined at the partition level, and several replicas can be configured under each partition, of which there can only be 1 leader replica and N-1 follower replicas. The producer writes messages to the partition, and the position information of each message in the partition is represented by a data called displacement, and the partition displacement always starts from 0.

• Record (message): The format of the message in Kakfa is as follows

If the key value of the message is empty when we send the message, Kafka will write the message to each partition of the current topic by polling by default.

If we specify the Key of the message, then the messages with the same key will be written into the same partition, so we can ensure that the messages with the same key are written in a certain order:

• What is the relationship between topics, partitions, replicas, and brokers?

• The first layer is the topic layer. Each topic can be configured with M partitions, and each partition can be configured with N replicas.
• The second layer is the partition layer. Only one of the N copies of each partition can act as the leader and provide external services; the other N-1 copies are follower copies, which only provide data redundancy.
• Different replicas belonging to the same partition will be distributed and stored on different Brokers to achieve high availability.
• The third layer is the message layer. The partition contains several messages, and the displacement of each message starts from 0 and increases sequentially.
• Finally, client programs can only interact with the leader replica of a partition.

---

How Kafka persists data

Kafka uses message logs to keep data. A log is a physical file on disk that can only append messages. The append write mechanism avoids slow random IO operations and replaces them with sequential IO operations with better performance. This is also an important means for Kafka to achieve high throughput.

Kafka divides the message log into multiple log segments, and the message is appended to the current latest log segment. When a log segment is full, Kafka will automatically split a new log segment and replace the old log segment. The log segment is sealed up, and the old log segment is regularly checked through the background scheduled task whether it can be deleted, so as to achieve the purpose of reclaiming disk space.

---

consumer

Kafka supports two message models, namely the point-to-point model and the publish-subscribe model. Kfaka implements these two consumption models by introducing the concept of consumer groups.

• Consumer Group
  • Multiple consumer threads that consume the same topic data can form a consumer group
  • A consumer group can subscribe to multiple topics and consume data under multiple topics
  • Multiple consumer instances form a group to consume a set of topics. Each partition in this group of topics will only be consumed by one consumer instance in the group, and other consumer instances in the group cannot consume it
    

Multiple consumer instances can consume at the same time, thereby accelerating the throughput (TPS) of the entire consumer. The consumer instance here can be a process running a consumer application, or a thread, and they are all called a consumer instance.

---

Partition copy data synchronization mechanism

Producers and consumers only communicate with the leader copy of the partition (primary partition copy), and the follower copy of the partition (partition copy) is responsible for synchronizing the data of the leader copy.

Here are some technical terms in Kafka:

• AR: Represents a collection of all replicas of a partition, including leader and follower replicas
• ISR: A set of partition replicas that are synchronized with the primary partition replica. Of course, there will also be a small amount of data delay here. Usually, what we call ISR also includes the Leader itself.
• OSR: The data synchronization status has been unable to keep up with the collection of primary partition replicas, which may be caused by problems such as the network
  

In order to ensure high availability (when the leader node hangs up, Kafka can still provide services), Kafka provides a partition copy replication mechanism. This copy is for the partition, so it can also be called a partition replica (partition replica). The number of replicas can be configured default.replication.factorby default value is 1, which means that the topic partition is not backed up. If it is configured as 2, it means that except for the leader node, there will be an additional copy for each partition in the topic.

If the leader of the partition copy fails, Kafka will select another one from the remaining partition copies as the leader to continue to provide services. This partition copy replication mechanism ensures that: if one or several servers hang up, the Kafka cluster still has partition copies available. Of course, if there is a copy of the Broker service that hangs up, the partition will hang up completely.

What kind of partition will be judged as OSR?

• Judging whether a follower partition copy is synchronized with the primary partition is achieved by judging the time span when the slave partition copy data lags behind the primary partition copy data. If this time span is greater than the parameter value configured by replica.lag.time.max.ms ( default 10 seconds), it is considered that the slave partition copy is not synchronized with the master partition copy, and the out-of-sync partition copy will be kicked out of the ISR set and put into the OSR set. Automatically returns to the ISR collection.

---

Can the unsynchronized replica in the OSR set be the leader?

The partition copy that is in the synchronization state with the primary partition copy is called ISR (including the Leader itself), and the data synchronization state has not kept up with the primary partition copy. The slave partition copy is called OSR.

Under normal circumstances, if the Leader hangs up, Kafka must elect a Leader from the ISR. But if the ISR list is empty, the leader can only be elected from the OSR. The function of the following parameter is to configure: whether to allow leader election from OSR.

unclean.leader.election.enable=false

The default value is true, we set it to false. Because when the leader is allowed to be elected from the OSR, and the leader is responsible for data communication with the client, the partition copy data in the OSR is seriously lagging and out of sync, which will lead to data loss.

Configuring this parameter means that we accept a situation: we would rather accept that the kafka service hangs up and not provide services; nor can we accept lingering services, which will eventually lead to data loss.

---

Minimum number of sync replicas

When Kafka producers send data, they can set a parameter called acks. If acks=all, it means that the broker can tell the producer that the message has been sent successfully only after all ISR partition copies have been successfully saved with persistent data.

• What are the problems with the above approach?
  
• If there are 3 partition copies in [1,2,3], 1 is the Leader partition copy, and the others are Follower partition copies.
• Due to network problems, the data synchronization progress of the [2,3] partition copy may be far behind the 1 partition copy, so now only the No. 1 partition copy itself is left in the ISR set.

Then the problem arises :

• When the producer acks=all, in this case only the No. 1 Leader partition copy is successfully persisted, and this message is sent successfully, because there is only 1 in the ISR set. What if the leader partition copy is also down at this time, is the data lost?
  • The data has indeed been sent out, and the reply producer data has been sent successfully
  • Leader hangs up by itself, and Follower has no synchronized data

How to solve this problem?

• When only the Leader is left in the ISR, the broker should not receive messages sent by the producer. It is better to accept the kafka service to hang up and not provide the service; nor to accept the lingering service, which will eventually lead to data loss.
• Set min.insync.replicas=n : When the producer sets acks to "all" (or "-1"), this configuration min.insync.replicasspecifies the minimum number of copies for write operations (that is, the minimum number of copies for data synchronization)
• The vernacular is: if the producer acks=all, when sending a message, the number of Broker's ISRs does not reach n, the Broker cannot receive this message, and needs to report an error directly to the producer:NotEnoughReplicas或NotEnoughReplicasAfterAppend
• A topic that contains 3 partition copies, if configured min.insync.replicas>=2, then when there is only one Leader partition copy left, the Leader partition copy becomes read-only ( can only provide consumption, cannot receive production data ). This can effectively avoid unexpected behaviors in data writing and reading during the Kafka topic partition replacement election process.

---

rebalance

Assuming that an instance in the consumer group is down, Kafka can automatically detect it, and then transfer the partition that the Failed instance was responsible for to other living consumers. This process is the notorious "rebalancing" in Kafka.

In the process of consuming messages, each consumer records where in the partition it consumes through the consumer displacement field.

The displacement here and the "displacement" of the partition in the message are not a concept. The displacement of the message in the partition indicates the position of the message in the partition, which is unchanged, that is, once the message is successfully written to a partition, Its displacement value is fixed.

The consumer displacement is different, it continues to move forward with the consumption progress of the consumer, and each consumer has its own consumer displacement, so it is necessary to distinguish the difference between these two types of displacement:

• Partition Offset: The offset of the message in the partition
• Consumer Displacement: Messenger Consumption Progress Indicator

---

Concept summary

• Message: Record
• Subject: Topic, a logical container for carrying messages, mainly used to distinguish specific services
• Partition: Partition, an ordered and unchanged message sequence, each topic can have multiple partitions
• Message offset: Offset, the position information of each message in the partition
• Copy: Replica, which is used to implement redundant storage of data. The copy is divided into leader copy and follower copy. The copy is under the partition level, that is, each partition can be configured with multiple copies to achieve high availability
• Producer: Produce , publishes messages to topics
• Consumer: Consumer , pull messages from topic
• Consumer displacement: Consumer Offser, which indicates the consumer's consumption progress, each consumer has its own consumer displacement
• Consumer group: Consumer Group, multiple consumer instances together form a group, consume multiple partitions at the same time to achieve high throughput
• Rebalance: Rebalance, after a consumer instance in the consumer group hangs up, other consumer instances automatically re-shard and subscribe to the topic partition. Rebalance is an important means for Kafka consumers to achieve high availability.
  

Source of this picture: Geek Time Kafka Core Technology and Practical Combat Lecture 2

---

Thinking: Why can't the follower replica in Kafka provide external read services like the slave nodes in the master-slave model?

• What scenarios are read-write separation suitable for?
  • Both Redis and Mysql support master-slave read-write separation, because usually there are more read operations than write operations for them. Therefore, the read-write separation architecture can be expanded horizontally by adding many followers to improve the performance of read operations.
  • As a message engine, Kafka provides read services instead of data storage. It usually involves frequent production and consumption of messages. At this time, the read and write operations are almost the same, and the read-write separation architecture cannot achieve much performance in this scenario. Lifting effect.
• What kind of choice did Kafka make in the scenario of writing more and reading less?
  • The Kafka copy mechanism uses asynchronous message pull, so there is inconsistency between the leader and the follower. If you want to adopt the read-write separation architecture, you must deal with the data consistency problem introduced by the copy asynchronous pull data, such as: read-your-writes , how to ensure monotonic reading and deal with the problem of inversion of the causal sequence of messages.
  • On the contrary, if read-write separation is not used, all client read-write requests are only processed on the leader, and there will be no such problems.
  • However, the problem of inversion of the global message order still exists in Kafak, and the simplest solution is to use a single partition.

Sources cited in the conclusions of this section:

Why does kafka not support master-slave separation?

---

Message Partitioning Mechanism

why partition

Kafka's message organization method is a three-level structure: topic-partition-message, each message under the topic will only be saved in a certain partition, and will not be saved in multiple partitions.

The main function of partitions is to provide load balancing capabilities, so as to achieve high scalability of the system. Different partitions can be placed on machines of different nodes, and data read and write operations are performed at the granularity of partitions, so that each node machine can independently execute the read and write request processing of its own partition. And, we can also increase the throughput of the overall system by adding new node machines.

Partitions can also be used to implement some business-level requirements, such as the issue of implementing business-level message order.

---

message routing policy

The message routing here is the algorithm for the producer to decide which partition to send the message to. Kafka provides us with a default partition strategy, which is the polling strategy, and also supports our custom partition strategy.

---

---

Kafka message model

• A partition is the smallest unit of parallelism
• A consumer can consume multiple partitions
• A partition can be consumed by consumers in multiple consumer groups
• But: A partition cannot be consumed by multiple consumers in the same consumer group at the same time

---

peer-to-peer model

The message sent by system A can only be received by system B, and no other system can read the message sent by A.

Kafka implements a point-to-point method, which can attribute all consumers to a consumer group, so that the message sent by the producer to the topic can only be consumed by one consumer in the consumer group subscribing to the topic:

---

publish-subscribe model

There may be multiple consumers sending messages to the same topic, and there may be multiple subscribers, and they can all receive messages on the same topic.

Kafka implements the publish-subscribe method, and can assign each consumer to a different consumer group, so that the message sent by the producer to the topic can be consumed by all consumers who subscribe to the topic:

---

message sequence

• production sequence
  
• For messages sent by the same producer to the same partition, the Offset sent first is smaller than the Offset sent later.
• The order of messages sent by the same producer to different partitions cannot be guaranteed.

---

• consumption order

• Consumers consume according to the order in which messages are stored in the partition
• Kafka only guarantees the order of messages within a partition, not the order of messages between partitions

---

How to guarantee message order

• Set up a partition so that the order of all messages can be guaranteed, but scalability and performance are lost
• Support by setting the key of the message, the message with the same key will be sent to the same partition

---

message passing semantics

• at most once — messages may be lost, never resent
• At least once - messages are not lost, but may be duplicated
• Exactly once — ensures that messages are delivered to the server without being repeated on the server

The above three message delivery semantics require producers and consumers to work together to complete.

---

• Producer at most once: After the producer sends the message, it is considered that the message is sent successfully
  

• The producer at least once: After the producer sends the message, if it does not receive the confirmation from the Broker, it will try again
  

• consumer at least once
  

• consumer at most once
  

• Exactly once: implemented after Kafka 0.11.0

---

producer

The Kafka producer client sends data as follows:

• The main thread calls KafkaProducer to send data, the data is not sent to the kafka broker server, but buffered first
• The asynchronous thread sender is responsible for sending the buffered data to the kafka broker server

• Using buffering can avoid server-side pressure caused by high concurrent requests, and buffering can also be used to send data in batches.
• The asynchronous sender thread is responsible for sending data, which avoids the main thread sending data blocking, causing delays in core business responses.

---

API use

• Introduce Kafka client dependencies

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>3.2.1</version>
        </dependency>

• Set producer client parameters

ProducerConfig: Set a series of configuration parameters for the producer client.

    private Properties props;

    @BeforeEach
    public void prepareTest(){
    
    
        props = new Properties();
        //kafka broker列表
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,KAFKA_ADDRESS);
        //可靠性确认应答参数
        props.put(ProducerConfig.ACKS_CONFIG,"1");
        //发送失败,重新尝试的次数
        props.put(ProducerConfig.RETRIES_CONFIG,"3");
        //生产者数据key序列化方式
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        //生产者数据value序列化方式
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    }

---

Message object – ProducerRecord

ProducerRecord is an entity object used by Kafka to encapsulate messages, which contains a lot of information, mainly including the following constructors:

 public ProducerRecord(String topic, Integer partition, Long timestamp, K key, V value, Iterable<Header> headers) {
    
    }
 public ProducerRecord(String topic, Integer partition, Long timestamp, K key, V value) {
    
    }
 public ProducerRecord(String topic, Integer partition, K key, V value, Iterable<Header> headers) {
    
    }
 public ProducerRecord(String topic, Integer partition, K key, V value) {
    
    }
 public ProducerRecord(String topic, K key, V value) {
    
    }
 public ProducerRecord(String topic, V value) {
    
    }

• topic: topic
• partition: The partition number of the topic, the number starts from 0, indicating which partition the message is sent to
• timestamp: timestamp, the default is the current timestamp
• key: The key of the message, which can be of different data types, but is usually a string type. Messages with the same key will be routed to the same partition, thus ensuring the order of the messages
• data: message data, can be different data types
• headers: message custom header information

---

Three ways to send data

The Kafka producer client has three ways to send messages:

• Only send regardless of the result (fire and forget): After the producer puts the message in the buffer, it thinks that the message is sent successfully and returns directly. Since Kafka is highly available, the message will be written successfully in most cases, but in Messages are lost in exceptional cases
• Synchronous send (sync send): Call the send method to return a Future object, use the get method of the Future object to block and wait for the message to be sent.
• Asynchronous send (async send): Provide a callback method when calling the send method, which will not block the current thread, and call back this method after receiving the Broker result

The following demonstrates the three message sending methods respectively.

You can set to open DEBUG or TRACE log to view the detailed log process of data sending

---

• The producer sends the message asynchronously: After the producer puts the message into the buffer, it considers that the message is sent successfully and returns directly

• KafkaProducer: producer object, used to send data
• ProducerRecord: Each piece of data must be encapsulated into a ProducerRecord object

    @Test
    public void sendAsyncWithNoCallBack() throws ExecutionException, InterruptedException {
    
    
        //ProducerRecord: 主题名,key,val
        sendMsgTemplate(data->new ProducerRecord<>(TEST_TOPIC,data,"val: "+data),null);
    }

    private void sendMsgTemplate(Function<String, ProducerRecord<String,String>> recordCallBack, Callback callback) {
    
    
        //try-with-resource -- 自动释放资源 -- 因为KafkaProducer实现了AutoCloseable接口
        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    
    
            for (int i = 0; i < 20; i++) {
    
    
                 //send方法参数: ProducerRecord<K, V> record, Callback callback
                 producer.send(recordCallBack.apply(Integer.toString(i)),callback);
            }
        }
    }

---

• Send message asynchronously, with callback method

    @Test
    public void sendAsyncWithCallBack() throws ExecutionException, InterruptedException {
    
    
        //ProducerRecord: 主题名,key,val
        sendMsgTemplate(data->new ProducerRecord<>(TEST_TOPIC,data,"val: "+data),((recordMetadata, e) -> {
    
    
             if(e==null){
    
    
                 System.out.println("消息发送成功,消息所在分区为:" + recordMetadata.partition()+" ,在分区中的位移为:"+recordMetadata.offset());
             }else{
    
    
                 e.printStackTrace();
             }
        }));
    }

The callback function will be called asynchronously before the producer receives the ack. This method has two parameters: RecordMetadata and Exception:

• If the message is sent successfully, the exception object is null and the message metadata object is not null
• If the message fails to be sent, the message metadata object is null and the exception object is not null

If the message fails to be sent, it will be automatically retried. There is no need to manually retry in the callback function. The number of retries is set by the parameter retries. After the number of retries reaches the upper limit, there will be an exception if the message still fails to be sent.

---

• Send messages synchronously

    @Test
    public void sendSync() throws ExecutionException, InterruptedException {
    
    
        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    
    
            Future<RecordMetadata> future = producer.send(new ProducerRecord<>(TEST_TOPIC, Integer.toString(520), "val=520"));
            RecordMetadata recordMetadata = future.get();
            if(recordMetadata!=null && recordMetadata.hasOffset()){
    
    
                System.out.println("消息同步发送成功: "+recordMetadata.offset());
            }
        }
    }

The return object of the send method of the producer is a Future type, so you can call the get() method of the Future object to block and wait for the response of the sending result, so as to achieve the effect of sending messages synchronously. The synchronization here means that after a message is sent, the current thread will be blocked until the ack message is returned.

---

• Send messages in bulk

    @Test
    public void sendBatch() throws InterruptedException, ExecutionException {
    
    
        //满足下面两个条件其中之一就会把消息缓冲区中的消息进行批量发送
        //每一批消息的大小: 默认是16KB
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);
        //延迟时间
        props.put(ProducerConfig.LINGER_MS_CONFIG, 1000);
        sendMsgTemplate(data -> {
    
    
            try {
    
    
                //模拟业务请求耗时,使得能够满足上面延迟时间条件,触发批量发送
                Thread.sleep(1000);
                return new ProducerRecord<>(TEST_TOPIC, data, "val: " + data);
            } catch (InterruptedException e) {
    
    
                throw new RuntimeException(e);
            }
        }, null);
    }

---

Producer Release Confirmation Mechanism

• acks parameter: optional values ​​are all, 1, 0, the default value is 1
  • 0 : fire and forget, that is, if I send it, it’s over, and I don’t care if the follow-up is successful or not. Under this setting, there is almost no guarantee for the high reliability of the message, but it has a huge throughput.
  • 1 : Writing to the master node is considered successful. This setting can guarantee high reliability and good throughput.
  • all / -1 : That is, writing to all copies in the ISR is considered successful. Under this setting, high reliability can be provided, but the throughput is relatively low.

props.put("acks",1);

• retries parameter: the number of producer retries, the default is 0

props.put("retries",0);

When acks and retries are used together, different messaging semantics can be generated:

• At most once: acks=0 or acks=1
• at least once: acks=-1 or akcs=all and retries>0

---

consumer

Partitions and Consumer Groups Review

A consumer group is a collection of consumers with common consumption characteristics, and these consumers jointly consume one or more topics.

Since a single consumer cannot satisfy the data processing speed under a certain topic, multiple consumers are required to load, which is an important reason for the emergence of consumer groups.

Each consumer in a consumer group has a consumer group ID. When creating a consumer, we can specify the group id to which the consumer belongs. If not specified, the default value is defined in the kafka安装目录/config/consumer.propertiesfile test-consumer-group.

Another point is repeatedly emphasized: a partition can only be consumed by one consumer in the consumer group.

• When the number of partitions of a topic is greater than the number of consumers in the consumer group subscribing to it, the following situations will occur:

The partition will be distributed to multiple consumers in the consumer group as evenly as possible

• When the number of partitions of a topic is less than the number of consumers in the consumer group subscribing to it, the following situations will occur:
  

There are four partitions and six consumers, and two consumers will be idle. Therefore, if the number of partitions does not match the number of consumers, creating
more consumers will not increase the data consumption rate.

• The best state is that the number of partitions is equal to the number of consumers
  

If we find that there is a backlog of consumption data for a certain topic, the first thing that comes to mind should be to optimize the consumer data consumption program and improve data processing efficiency. If the demand is still not met, then simultaneously increase the number of topic partitions and the The number of consumers, so that the two are consistent.

• Multiple topics, multiple consumer groups

• A partition can only be consumed by one consumer in the consumer group
• A consumer may consume multiple partitions within a topic
• The best state is that the number of partitions of the topic is equal to the number of consumers in the consumer group
• If the number of consumers in a consumer group is greater than the number of partitions of the topic it subscribes to, extra consumers will idle

---

Consumer group data backlog view and processing

Therefore, to check the progress of data consumption, or whether there is a backlog of message data, it is checked in units of "consumer" groups.

You can view the consumption progress of a consumer group through the command line as follows

./kafka-consumer-groups.sh --bootstrap-server kafka-0:9092 --describe --group dhy-group

The format of the response result is as follows:

TOPIC    PARTITION    CURRENT-OFFSET    LOG-END-OFFSET    LAG
topic2     0            241019           395308           154289          
topic2     1            241019           398817           157799
topic1     0            854144           855809           1669    

• One partition information of one topic occupies one line
• The consumer group subscribes to three topics: topic1 (one partition 0), topic2 (two partitions 0, 1)
• CURRENT-OFFSET is the offset of the current data consumption progress
• LOG-END-OFFSET is the data information offset that the current partition has received
• LAG = (LOG-END-OFFSET - CURRENT-OFFSET). The larger the value of LAG, the more serious the data backlog.

---

How do you think about data backlogs?

• If a partition has an average of 2,000 data per day, and the backlog is stable at 10,000, and has no effect on the normal use of users, then it is not considered a data backlog. If the LAG soars to hundreds of thousands under abnormal conditions, then this is the data backlog.

How to solve the data backlog?

• Optimize consumer business logic and provide data processing speed
• Increase the number of partitions of the topic where the backlog occurs, and at the same time increase the number of consumers consuming the topic partition, so that it is equal to the number of partitions of the topic
• Flow control with the help of flow limiting tools such as sentinel

The command to increase the number of topic partitions is as follows:

./kafka-topics.sh --alter \
--bootstrap-server kafka-0:9092,kafka-1:9092,kafka-2:9092 \
--topic test2
--partitions 4

Note: The number of partitions for a topic can only be increased, not decreased.

---

API use

review:

• There is a topic _consumer_offsets in Kafka, which is used to keep which topic the consumer consumes and which consumption position of which partition, so that once a consumer restarts, it can quickly restore to the last consumption position.
• After the consumer gets the message, it will store the message location under the partition of the topic _consumer_offsets, and when it reads next time, it will return the next consumption location.

• Introduce Kafka client dependencies

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>3.2.1</version>
        </dependency>

• Set consumer client parameters

public class KafkaConsumerTest {
    
    
    private static final String TEST_TOPIC = "test1";
    private Properties props;

    @BeforeEach
    public void prepareTest() {
    
    
        props = new Properties();
        //kafka集群信息
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, KAFKA_ADDRESS);
        //消费者组名称
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "dhy_group");
        //key的反序列化器
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        //value的反序列化器
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    }
}

• Set the template method for the consumer to consume messages

    /**
     *  recordConsumer针对单条数据进行处理，此方法中应该做好异常处理，避免外围的while循环因为异常中断。
     */
    public void consumeTemplate(Consumer<ConsumerRecord<String,String>> recordConsumer,Consumer<KafkaConsumer<String,String>> afterCurrentBatchHandle){
    
    
        //1.创建消费者
        KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
        //2.订阅Topic--支持正则表达式: consumer.subscribe("test.*");
        consumer.subscribe(Collections.singletonList(TEST_TOPIC));
        try {
    
    
            while (true) {
    
    
                //循环拉取数据,
                //  Duration超时时间，如果有数据可消费，立即返回数据
                // 如果没有数据可消费，超过Duration超时时间也会返回，但是返回结果数据量为0
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(100));
                for (ConsumerRecord<String, String> record : records) {
    
    
                        recordConsumer.accept(record);
                }
               if (afterCurrentBatchHandle != null) {
    
    
                    afterCurrentBatchHandle.accept(consumer);
               }
            }
        } finally {
    
    
            //退出应用程序前使用close方法关闭消费者，
            // 网络连接和socket也会随之关闭，并立即触发一次再均衡
            consumer.close();
        }
    }

    private static void printRecord(ConsumerRecord<String, String> record) {
    
    
        System.out.println("topic:" + record.topic()
                + ",partition:" + record.partition()
                + ",offset:" + record.offset()
                + ",key:" + record.key()
                + ",value" + record.value());
        record.headers().forEach(System.out::println);
    }

---

consumption offset

Each consumer client of Kafka consumes the data of one partition, and at the same time uses the message offset to identify the current consumption progress, which is also called the consumer offset (Consumer Offset):

• For a consumer group, the record is the consumption progress of the consumer group in multiple partitions, that is, a set of <K, V> pairs, Key is the partition object, and V corresponds to the latest offset of the consumer consuming the partition
• In the figure above, yellow represents unconsumed message data in the partition, and green represents message data that has been consumed. The consumer offset is the offset of the next message to be consumed, not the offset of the currently consumed message.
• The message data consumed by the consumer will be submitted with an offset, so that after the consumer fails and restarts, it can read the offset previously submitted by the consumer group from Kafka, and then continue to consume from the corresponding offset.

---

HighWatermark

HighWatermark is referred to as HW for short, which stands for high water mark, and messages after high water mark and high water mark have actually physically existed in the partition, but cannot be consumed.

• The offset of HighWatermark is greater than the offset of Consumer Offset
• Log End Offset (LEO) refers to the offset of the last message to be appended to the current copy

What is the role of HW? Why can't the messages between HW and LEO be consumed?

• First, the data of the Follower copy is synchronized from the Leader copy. Kafka's producers and consumers only communicate with the Leader copy.
• Secondly, due to the data synchronization delay between the Follower copy and the Leader copy, there must be a certain moment when the data writing progress of the Follower copy lags behind that of the Leader copy.
• So HW represents the log offset that all ISR copies of the partition can reach synchronously , as shown in the following figure:

The message of not allowing HW and subsequent offsets to be consumed is to avoid switching to Follower2 when the partition leader is re-elected, and the progress of consumption data cannot be synchronized.

---

Automatic submission and manual submission (synchronous submission and asynchronous submission)

Consumer needs to submit the offset of a certain partition it consumes to the Broker. The offset submission method is divided into automatic submission and manual submission. From the perspective of whether it is blocked, it can be divided into synchronous submission and asynchronous submission.

The following consumer demo use cases will use the consumer consumption template method given above

• Automatic submission—consumption at most once : When we set the parameter of the client ENABLE_AUTO_COMMITto true, the consumer will automatically submit the offset for us periodically, and the submission interval is AUTO_COMMIT_INTERVAL_MScontrolled by the parameter.

    @Test
    public void consumeWithAutoCommit(){
    
    
        //下面两个属性用于设置自动提交---只要消息被拉取到,并且到了指定的间隔时间,消息便会自动提交
        props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
        props.setProperty(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
        consumeTemplate(KafkaConsumerTest::printRecord,null);
    }

Notice:

• Since the consumer is single-threaded, in actual situations, it may not AUTO_COMMIT_INTERVAL_MSsubmit the offset every other time. The specific execution process is as follows:
  • Get a batch of messages through poll and then consume them
  • When polling the next batch of data, judge whether the time interval of the last poll data is greater than AUTO_COMMIT_INTERVAL_MS, if greater, automatically submit the offset
  • The offset automatically submitted here is the offset offset of the data consumed in the last batch

• Manual synchronous submission (batch submission) - consume at least once : the automatic submission method, the offset can only be submitted when the poll method is called, if we want to submit the offset anywhere in the program processing, we need to consider using manual way to submit.
  • Set the client ENABLE_AUTO_COMMITparameter to false
  • Use commitSync to complete the manual synchronous commit of the offset

    @Test
    public void consumeWithNoAutoCommitWithSyncBatch(){
    
    
        //设置为手动提交--默认值
        props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        //缓冲记录,用于批量手动提交
        List<ConsumerRecord<String,String>> buffer=new ArrayList<>();
        final int minBatchSize=3;
        consumeTemplate(buffer::add, consumer->{
    
    
            if(buffer.size()>=minBatchSize){
    
    
                //数据操作
                buffer.forEach(KafkaConsumerTest::printRecord);
                //批量提交
                consumer.commitSync();
                //清空缓存
                buffer.clear();
            }
        });
    }

Notice:

• commitSync is a synchronous method that blocks until the offset is successfully committed
• The commitSync synchronous submission will be retried after the failure, and if the retry still fails, a CommitFailedException will be thrown
• The commitSync synchronization method will block the consumption thread, so try to avoid using it for business scenarios that require high message consumption speed.

• Manual asynchronous submission (batch submission) - consume at least once : using asynchronous submission usually uses commitAsync with a callback function, if the offset submission fails, log or exception handling

    @Test
    public void consumeWithNoAutoCommitWithAsyncBatch(){
    
    
        //设置为手动提交--默认值
        props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        //缓冲记录,用于批量手动提交
        List<ConsumerRecord<String,String>> buffer=new ArrayList<>();
        final int minBatchSize=3;
        consumeTemplate(buffer::add, consumer->{
    
    
            if(buffer.size()>=minBatchSize){
    
    
                //数据操作
                buffer.forEach(KafkaConsumerTest::printRecord);
                //批量提交
                consumer.commitAsync(new OffsetCommitCallback() {
    
    
                    @Override
                    public void onComplete(Map<TopicPartition, OffsetAndMetadata> offsets, Exception exception) {
    
    
                             if(exception!=null){
    
    
                                 //异常处理
                             }
                    }
                });
                //清空缓存
                buffer.clear();
            }
        });
    }

Notice:

• The problem with commitAsync is that it doesn't automatically retry when the commit offset fails

• Synchronous submission combined with asynchronous submission : staged manual submission, in order to avoid blocking, call the commitAsync asynchronous submission method, once the consumer thread is abnormal, call the commitSync method to perform synchronous blocking submission, to ensure that the offset can be successfully submitted before the consumer is closed

    @Test
    public void consumeWithNoAutoCommitCombineSyncBatchAndAsyncBatch() {
    
    
        KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
        consumer.subscribe(Collections.singletonList(TEST_TOPIC));
        try {
    
    
            while (true) {
    
    
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(100));
                records.forEach(KafkaConsumerTest::printRecord);
                consumer.commitAsync();
            }
        } catch (CommitFailedException e) {
    
    
            //记录错误日日志,进行告警处理
            e.printStackTrace();
        } finally {
    
    
            consumer.commitSync();
            consumer.close();
        }
    }

---

Double consumption problem

• Under what circumstances will repeated consumption occur?
  • Suppose we pull 500 messages at one time, but when the Consumer consumes the 200th message, it crashes and restarts
  • Since the previous 199 messages have been consumed, but the consumption displacement has not been submitted, the 500 data from the previous time will still be pulled when consuming again, which will cause the messages in the range of 0~199 to be consumed repeatedly
    

Notice:

• As long as the data is consumed in batches and the offset is submitted in batches, the problem of repeated consumption cannot be avoided. It cannot be submitted manually or automatically, whether synchronously or asynchronously.

The easiest way to avoid repeated consumption is to submit the offset of this message every time a message is consumed.

• Manual commit - single commit

    @Test
    public void consumeWithNoAutoCommitWithSingle() {
    
    
        //设置为手动提交--默认值
        props.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList(TEST_TOPIC));
        Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
        while (true) {
    
    
            //一次性请求尽可能多的数据
            ConsumerRecords<String, String> records = consumer.poll(Long.MAX_VALUE);
            for (ConsumerRecord<String, String> record : records) {
    
    
                 printRecord(record);
                 //记录下offsets信息
                 offsets.put(new TopicPartition(record.topic(),record.partition()),new OffsetAndMetadata(record.offset()+1));
                 //当前消息处理完,就提交当前消息的偏移量
                 consumer.commitAsync(offsets,null);
            }
            try {
    
    
                //处理完当前批次的消息,在拉取下一批消息前,调用commitSync方法提交当前批次最新消息
                consumer.commitSync(offsets);
            }catch (CommitFailedException e){
    
    
                e.printStackTrace();
            }
        }
    }

Notice:

• Although the single submission method can avoid repeated consumption of messages, the efficiency is very low, so it is recommended to use batch consumption
• The best way to avoid repeated consumption of messages is to ensure the robustness of the consumer-side program, fully test it, avoid exceptions due to data format and other issues, and make alarms and log records if an exception occurs

---

data loss problem

Usually, we submit the offset of this batch of data after processing the current batch of messages, so as long as the exception is handled properly, there is no problem of data loss.

However, in some scenarios, there is still a risk of data loss, as shown in the following figure:
The Consumer removes 300 pieces of data at a time, and then transfers the message to a separate thread pool for processing, and then the main thread continues to execute and submit The offset of these 300 consumptions.

Assume that there are three threads in the thread, and each thread is responsible for processing one hundred messages, but an exception occurs when thread 2 and thread 3 process the messages. Since the main thread has submitted the offset, the data that the child thread failed to execute, It will never be consumed, which creates a data loss problem.

If you want to avoid these problems, don't use threads to process message data, because the multiple consumers contained in the consumer group are themselves multi-threaded, so don't start multiple threads in the consumer code.

---

Specify partition offset for consumption

Sometimes due to some failure reasons, it may be necessary to perform data remedial measures. At this time, it is necessary to start consumption from the specified offset for a certain topic or partition.

• seek specifies to consume a certain topic partition, and specifies offset (long) to start consumption
• seekToBeginning starts consumption from the queue head (offset=0) of a topic partition
• seekToEnd starts consumption from the tail of the queue (offset = offset of the latest piece of data) in a topic partition.

For example:

        //指定消费者消费主题及分区
        String topic="foo";
        TopicPartition partition0 = new TopicPartition(topic, 0);
        TopicPartition partition1 = new TopicPartition(topic, 1);
        consumer.assign(Arrays.asList(partition0,partition1));
        //指定从foo主题的partition0分区的offset=1的位置开始消费
        consumer.seek(partition0,1);        

---

• auto.offset.resetParameter meaning: How to start a new consumption when there is no offset submitted by the consumer or the offset specified by the program does not exist in each partition of the server.

props.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");

This parameter is also an important configuration parameter specifying consumption from a specified location. This parameter has three values:

earliest、latest、none

Note: There are two situations in which there are no offsets submitted by consumers

• It is possible that the partition of this topic is newly created and no consumer has consumed it before
• It is possible that the consumer group is newly created, and this consumer group has not consumed this partition before

Parameter value analysis:

• earliest (default): When there is a submitted offset under each partition, consumption starts from the submitted offset; when there is no submitted offset, consumption starts from the beginning.
• latest: When there is a submitted offset under each partition, consumption starts from the submitted offset; when there is no submitted offset, the latest data generated under the partition is consumed.
• none: When there is a committed offset in each partition of the topic, consumption starts after the offset; as long as there is a partition without a committed offset, an exception is thrown

---

summary

This article is organized as a study note for the introduction to Kafak, focusing on the installation process of Kafka (to be supplemented and improved), the core concepts of Kafka and the simple API usage of Kafka producers and consumers.

In the follow-up Kafak basics, some advanced knowledge of Kakfa will be sorted out, and the use of SpringBoot to integrate Kafka will be included.